SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation

2025-10-13

AI Summary

Evaluating multi-turn interactive agents typically relies on human annotators, resulting in low efficiency and poor scalability; existing simulated user approaches lack domain adaptability and fail to replicate authentic user behavior. Method: This paper proposes a domain-knowledge-integrated user simulation framework that jointly models top-down business logic (e.g., user personas, goal constraints) and bottom-up empirical data (e.g., product catalogs, FAQs), enabling a knowledge-enhanced large language model–driven simulator with integrated business rule modeling, intent generation, and dialogue state tracking. Contribution/Results: Experiments demonstrate substantial improvements in dialogue authenticity and diversity. In defect detection tasks, the proposed simulator identifies 33% more agent errors than baseline methods, significantly enhancing the accuracy and controllability of automated evaluation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Evaluating multi-turn interactive agents is challenging due to the need for human assessment. Evaluation with simulated users has been introduced as an alternative, however existing approaches typically model generic users and overlook the domain-specific principles required to capture realistic behavior. We propose SAGE, a novel user Simulation framework for multi-turn AGent Evaluation that integrates knowledge from business contexts. SAGE incorporates top-down knowledge rooted in business logic, such as ideal customer profiles, grounding user behavior in realistic customer personas. We further integrate bottom-up knowledge taken from business agent infrastructure (e.g., product catalogs, FAQs, and knowledge bases), allowing the simulator to generate interactions that reflect users' information needs and expectations in a company's target market. Through empirical evaluation, we find that this approach produces interactions that are more realistic and diverse, while also identifying up to 33% more agent errors, highlighting its effectiveness as an evaluation tool to support bug-finding and iterative agent improvement.

Problem

• Simulating realistic user behavior for multi-turn agent evaluation
• Integrating domain-specific knowledge from business contexts into simulations
• Enhancing agent error detection through knowledge-grounded user interactions

Innovation

• Integrates top-down business logic for user personas
• Leverages bottom-up business infrastructure knowledge bases
• Generates realistic diverse interactions to detect agent errors