End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering

2026-03-11

AI Summary

This work addresses the challenge of hallucinated or factually incorrect responses commonly generated by domain-specific chatbots, for which existing evaluation approaches—relying on manual annotation or static metrics—lack scalability. The authors propose a modular, language-agnostic, end-to-end automated evaluation framework that automatically generates question-answer pairs from a knowledge base, leverages large language models to compare system outputs against reference answers, and incorporates confidence calibration with uncertainty-aware filtering to adaptively flag low-confidence predictions, thereby minimizing human intervention. Experiments on a Vietnamese news dataset demonstrate strong alignment between the proposed method and human judgments, achieving high scalability and cross-lingual applicability while substantially reducing annotation costs.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) combined with retrieval augmented generation have enabled the deployment of domain-specific chatbots, but these systems remain prone to generating unsupported or incorrect answers. Reliable evaluation is therefore critical, yet manual review is costly and existing frameworks often depend on curated test sets and static metrics, limiting scalability. We propose an end-to-end automatic evaluator designed to substantially reduce human effort. Our system generates Q\&A pairs directly from the underlying knowledge base, uses LLMs to judge chatbot responses against reference answers, and applies confidence-based filtering to highlight uncertain cases. Applied to a Vietnamese news dataset, the evaluator achieves high agreement with human judgments while significantly lowering review overhead. The framework is modular and language-agnostic, making it readily adaptable to diverse domains. This work introduces a practical, scalable solution for evaluating chatbots with minimal reliance on manual intervention.

Problem

• chatbot evaluation
• large language models
• retrieval augmented generation
• automatic evaluation
• uncertainty filtering

Innovation

• end-to-end evaluation
• adaptive reasoning
• uncertainty filtering
• retrieval-augmented generation
• automatic QA generation