Towards Automated Error Discovery: A Study in Conversational AI

2025-09-13

AI Summary

Large language model (LLM)-based dialogue agents often suffer from latent, previously unseen errors—e.g., those induced by model updates or shifts in user behavior—that evade conventional monitoring. Method: This paper proposes SEEED, a representation-learning-based framework for automated error discovery. Its core innovation lies in an enhanced soft nearest-neighbor loss, augmented with label-guided sample ranking, and an encoder architecture integrating soft clustering expansion, distance-weighted negative sample mining, and contrastive learning. Contribution/Results: SEEED significantly improves generalization to novel error types and emerging user intents. Evaluated on multiple annotated dialogue datasets, it outperforms strong baselines—including GPT-4o and Phi-4—achieving up to an 8-percentage-point gain in unknown error detection accuracy. The framework provides a scalable, robust solution for production-grade LLM agent monitoring and deployment reliability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Although LLM-based conversational agents demonstrate strong fluency and coherence, they still produce undesirable behaviors (errors) that are challenging to prevent from reaching users during deployment. Recent research leverages large language models (LLMs) to detect errors and guide response-generation models toward improvement. However, current LLMs struggle to identify errors not explicitly specified in their instructions, such as those arising from updates to the response-generation model or shifts in user behavior. In this work, we introduce Automated Error Discovery, a framework for detecting and defining errors in conversational AI, and propose SEEED (Soft Clustering Extended Encoder-Based Error Detection), as an encoder-based approach to its implementation. We enhance the Soft Nearest Neighbor Loss by amplifying distance weighting for negative samples and introduce Label-Based Sample Ranking to select highly contrastive examples for better representation learning. SEEED outperforms adapted baselines -- including GPT-4o and Phi-4 -- across multiple error-annotated dialogue datasets, improving the accuracy for detecting unknown errors by up to 8 points and demonstrating strong generalization to unknown intent detection.

Problem

• Detecting unspecified errors in conversational AI systems
• Identifying errors from model updates and user shifts
• Improving automated error discovery with encoder-based methods

Innovation

• Encoder-based error detection framework
• Amplified distance weighting for negatives
• Label-based sample ranking for contrast