Convert Language Model into a Value-based Strategic Planner

2025-05-11

AI Summary

This paper addresses the persistent challenge of low long-term user satisfaction in emotional support conversations (ESC). Existing approaches lack explicit dialogue state modeling, hindering optimization of sustained emotional relief. To bridge this gap, we propose *straQ**, the first framework to integrate Q-learning into LLM-based dialogue planning. *straQ* explicitly models dialogue state transitions and long-term reward accumulation, jointly leveraging state-action value estimation and policy-guided decoding to enable autonomous, plug-and-play strategic response generation by LLMs. Crucially, it balances immediate empathic responsiveness with long-horizon goal alignment. Empirical evaluation across multiple ESC benchmarks demonstrates that *straQ* significantly outperforms strong baselines—including direct prompting, chain-of-thought reasoning, fine-tuning, and finite-state-machine approaches—on both long-term satisfaction metrics and strategic consistency.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Emotional support conversation (ESC) aims to alleviate the emotional distress of individuals through effective conversations. Although large language models (LLMs) have obtained remarkable progress on ESC, most of these studies might not define the diagram from the state model perspective, therefore providing a suboptimal solution for long-term satisfaction. To address such an issue, we leverage the Q-learning on LLMs, and propose a framework called straQ*. Our framework allows a plug-and-play LLM to bootstrap the planning during ESC, determine the optimal strategy based on long-term returns, and finally guide the LLM to response. Substantial experiments on ESC datasets suggest that straQ* outperforms many baselines, including direct inference, self-refine, chain of thought, finetuning, and finite state machines.

Problem

• Convert LLMs into value-based strategic planners for emotional support
• Address suboptimal long-term satisfaction in emotional support conversations
• Optimize ESC strategies using Q-learning on large language models

Innovation

• Leverage Q-learning on large language models
• Propose plug-and-play straQ* framework
• Optimize strategy based on long-term returns