Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue

2026-08-11

AI Summary

This work addresses the challenge that existing large language models struggle to consistently track users’ dynamically evolving emotions in multi-turn empathetic dialogues, and conventional reinforcement learning suffers from a mismatch between policy and experience due to fixed interaction distributions. To overcome these limitations, the authors propose a dual-loop self-evolution framework: an inner loop optimizes the empathetic policy using verifiable emotion-based rewards, while an outer loop dynamically reshapes the interaction experience distribution based on policy performance, enabling co-evolution of policy and data. The approach innovatively integrates boundary-capability-prioritized sampling, uncertainty-guided exploration, and uniform replay mechanisms to achieve sample-efficient training under limited rollout budgets. Evaluated on the SAGE benchmark, the method boosts Qwen3-8B’s performance from 53.87 to 79.24, substantially outperforming protocol-matched baselines by +7.23 points.

Abstract

Large language models have demonstrated conversational capabilities, yet empathetic competence remains challenging. Empathetic support is inherently multi-turn and path-dependent: users disclose concerns gradually, emotions evolve over time, and early responses shape trust and receptivity. Reinforcement learning with verifiable emotion rewards provides scalable supervision for long-horizon interactions. However, existing methods evolve the dialogue policy while keeping its training interaction distribution fixed, creating a mismatch between policy competence and training experience. We introduce a dual-loop self-evolution framework driven by verifiable emotion feedback. With the user simulator and verifier frozen, the inner loop optimizes the multi-turn policy using continuous emotion rewards, while the outer loop uses the same outcomes to estimate policy-relative interaction utility and adapt experience. To obtain estimates from sparse, stochastic rollouts, the framework holds the scenario and interaction state constant within each group and prioritizes conditions whose group pass rates lie near the policy's competence boundary. A hierarchical controller shares evidence across support intents, while uncertainty-guided exploration and uniform rehearsal prevent premature exclusion. The resulting distribution generates trajectories, closing both loops without increasing the rollout budget. On SAGE, our framework raises Qwen3-8B Overall from 53.87 to 79.24 and outperforms protocol-matched uniform emotion-reward reinforcement learning by 7.23 points.

Problem

• empathetic dialogue
• multi-turn interaction
• emotion evolution
• policy-training mismatch
• verifiable feedback

Innovation

• dual-loop self-evolution
• verifiable emotion feedback
• multi-turn empathetic dialogue
• policy-relative interaction utility
• uncertainty-guided exploration