LLM-guided Plan and Retrieval: A Strategic Alignment for Interpretable User Satisfaction Estimation in Dialogue

2025-03-06

AI Summary

Existing user satisfaction estimation (USE) methods suffer from ambiguous attribution, high annotation costs, poor interpretability, and low inference efficiency. To address these challenges, we propose PRAISE—a novel, three-tier interpretable framework comprising *Strategy Planning*, *Feature Retrieval*, and *Score Analysis*. It leverages natural-language strategies to guide intent alignment, achieving both high prediction accuracy and instance-level attribution explanations. Crucially, PRAISE fully decouples large language models (LLMs) from the inference pipeline via knowledge distillation, semantic retrieval, and multi-stage collaborative modeling—enabling efficient, LLM-free deployment. Evaluated on three standard USE benchmarks, PRAISE achieves state-of-the-art performance, delivers fine-grained attribution analysis, reduces inference latency by 42%, and operates without runtime LLM invocation.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Understanding user satisfaction with conversational systems, known as User Satisfaction Estimation (USE), is essential for assessing dialogue quality and enhancing user experiences. However, existing methods for USE face challenges due to limited understanding of underlying reasons for user dissatisfaction and the high costs of annotating user intentions. To address these challenges, we propose PRAISE (Plan and Retrieval Alignment for Interpretable Satisfaction Estimation), an interpretable framework for effective user satisfaction prediction. PRAISE operates through three key modules. The Strategy Planner develops strategies, which are natural language criteria for classifying user satisfaction. The Feature Retriever then incorporates knowledge on user satisfaction from Large Language Models (LLMs) and retrieves relevance features from utterances. Finally, the Score Analyzer evaluates strategy predictions and classifies user satisfaction. Experimental results demonstrate that PRAISE achieves state-of-the-art performance on three benchmarks for the USE task. Beyond its superior performance, PRAISE offers additional benefits. It enhances interpretability by providing instance-level explanations through effective alignment of utterances with strategies. Moreover, PRAISE operates more efficiently than existing approaches by eliminating the need for LLMs during the inference phase.

Problem

• Improves user satisfaction estimation in dialogue systems
• Addresses challenges in understanding user dissatisfaction reasons
• Reduces annotation costs for user intention analysis

Innovation

• LLM-guided strategy planning for user satisfaction
• Retrieval of relevance features from utterances
• Efficient inference without LLMs for cost reduction