TLDR: The research introduces Automated Error Discovery, a framework and its implementation SEEED (Soft Clustering Extended Encoder-Based Error Detection), designed to detect and define both known and previously unknown errors in conversational AI. SEEED uses LLMs for dialogue summarization, encoder-based soft clustering with enhanced loss functions and a novel sampling strategy (LBSR) for error detection, and then generates definitions for new error types. It significantly outperforms baselines like GPT-4o and Phi-4 in identifying novel errors and generalizes well to unknown intent detection, making conversational AI more robust.
Conversational AI agents, powered by large language models (LLMs), have become incredibly fluent and coherent in their interactions. However, they still produce unexpected and undesirable behaviors, commonly known as errors. These errors are challenging to prevent from reaching users, especially as AI models are updated or user behaviors shift, leading to new types of errors that current LLMs struggle to identify if not explicitly programmed to do so.
A new research paper, “Towards Automated Error Discovery: A Study in Conversational AI” by Dominic Petrak, Thy Thy Tran, and Iryna Gurevych, introduces a novel framework called Automated Error Discovery. This framework aims to detect and define errors in conversational AI, including those that are entirely new or unknown to the system. At the heart of this framework is SEEED (Soft Clustering Extended Encoder-Based Error Detection), an innovative approach designed to implement this discovery process.
How SEEED Works
SEEED operates through several key components to achieve its goal of automated error discovery:
-
Summary Generation: First, an LLM is prompted to create a concise summary of the dialogue context. This summary focuses on information that might indicate an error in the agent’s most recent response. This helps to distill complex conversations into manageable insights relevant for error identification.
-
Error Detection: The dialogue context and its summary are then processed by specialized Transformer-based encoders. These encoders generate a combined representation, which is then fed into a soft clustering algorithm called NNK-Means. Unlike traditional ‘hard’ clustering, where each data point belongs to only one group, soft clustering allows a dialogue to belong to multiple error categories simultaneously, providing a more nuanced understanding of complex errors.
-
Enhanced Learning: To make the error detection more robust, SEEED incorporates two key enhancements. It uses an improved version of the Soft Nearest Neighbor Loss, which helps in refining decision boundaries by giving more weight to ‘negative’ examples (dialogues that are different from the current one). Additionally, it introduces Label-Based Sample Ranking (LBSR), a smart sampling strategy that selects highly contrasting examples during training. This ensures the model learns to distinguish between error types more effectively.
-
Error Definition Generation: When SEEED identifies an error type that isn’t among its known categories, it doesn’t just stop there. It leverages another LLM (Llama-3.1 8B-Instruct) to generate a clear and concise definition for this newly discovered error. This process is guided by the summaries of related dialogues and examples of existing error definitions, ensuring consistency and relevance.
Also Read:
- Navigating LLM Sensitivity: A New Approach to Testing Response Differences
- Detecting AI Hallucinations Through Internal Reasoning
Performance and Impact
The research demonstrates that SEEED significantly outperforms existing methods, including powerful LLMs like GPT-4o and Phi-4, in detecting novel error types. Across various error-annotated dialogue datasets (FEDI-Error, Soda-Eval, and ABCEval), SEEED improved the accuracy for identifying unknown errors by up to 8 points. This highlights its strong ability to generalize and adapt to new error patterns without explicit prior instruction.
Beyond error detection, SEEED also shows impressive generalization capabilities in related tasks, such as intent detection. It achieved up to a 17-point improvement in accuracy for identifying unknown user intents compared to state-of-the-art methods, producing more compact and well-separated clusters of intents.
The study also includes ablation experiments, which confirm the critical contribution of each component of SEEED—from the soft clustering algorithm and the enhanced loss function to the Label-Based Sample Ranking and the use of dialogue summaries. These findings underscore the thoughtful design behind the framework.
This work marks a significant step towards making conversational AI agents more reliable and trustworthy by enabling them to autonomously discover and understand new types of undesirable behaviors. For more details, you can read the full paper here.
