A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models

2026-09-01

AI Summary

本文提出一种三代理框架，通过模拟对话评估和改进大型语言模型在处理模糊问题时的澄清能力，以提高交互系统的用户意图理解准确性。

Abstract

Large Language Models (LLMs) are increasingly deployed in interactive systems where understanding user intent precisely is paramount. A key capability for such systems is effective question clarification, especially when user queries are ambiguous or underspecified. This paper introduces a novel tri-agent framework for the robust evaluation of an LLM's ability to engage in clarifying dialogue. Our framework comprises three distinct LLM-based agents: (1) a Question Clarifying Agent (QCA), the system under evaluation, tasked with identifying ambiguities and posing clarifying questions; (2) a Respondent Agent (RA), designed to simulate human user responses, potentially including irrelevant or challenging replies; and (3) an Evaluator Agent (EA), an LLM-as-a-judge, which assesses the quality of the dialogue based on a comprehensive set of metrics. We detail a methodology for synthetic data generation in the supply chain domain as an example. We propose metrics evaluating ambiguity handling, question quality, dialogue efficiency, language appropriateness, and final intent alignment. We also briefly discuss the validation of the EA against human judgments. This work provides a structured approach to benchmark, validate, and improve the clarification capabilities of conversational LLM applications.

Problem

• Large Language Models
• Question Clarification
• User Intent
• Ambiguity
• Interactive Systems

Innovation

• Tri-Agent Framework
• Question Clarification
• Large Language Models
• Dialogue Evaluation
• Ambiguity Handling