ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

2026-08-22

AI Summary

本文提出了一种双层强化学习框架ToSCA，通过结合DQN和PPO算法及双重粒度奖励机制，解决了对话系统中策略确定与响应质量的问题。

Abstract

Humans have multiple levels of temporal abstractions on daily interaction and thinking, such as concept perception and strategic planning. Inspired by this nature, we propose a two-level hierarchical reinforcement learning (RL) framework for conversational agents, bridging the gap between previous token-level or utterance-level RL methods. Developed on a two-level MDP, the token-level response decoding is conditioned on the utterance-level action, the explicit textual strategies. Based on theoretical derivation and efficiency consideration, we use DQN to solve the high-level critic and PPO to solve the low-level actor-critic. To further alleviate the reward sparsity and facilitate the convergence, we also design the dual-granularity reward mechanism, in which the utterance-level satisfaction score is integrated with token-level intrinsic motivation and K-L penalty. Experiments on both daily and emotional support conversations show that our method outperforms versatile baselines in strategy determination and response quality. Our implementation is available at https://github.com/AaronJi/ToSCA.

Problem

• Hierarchical Reinforcement Learning
• Temporal Abstractions
• Strategic Abstractions
• Conversational Agents

Innovation

• Hierarchical Reinforcement Learning
• Temporal Abstractions
• Dual-granularity Reward Mechanism
• DQN and PPO