Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

2026-08-18

AI Summary

为解决多轮用户交互中信用分配问题，提出FACA方法利用下一轮用户反馈提供局部信用信号，改进了多轮对话代理性能。

Abstract

User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce \textbf{F}eedback-\textbf{A}ware \textbf{C}redit \textbf{A}ssignment (\textsc{FACA}), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, \textsc{FACA} improves the nine-domain $τ$-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents.

Problem

• multi-turn user interaction
• interactive reinforcement learning
• user feedback
• credit assignment
• dialogue and tool coordination

Innovation

• FACA
• multi-turn interaction
• user feedback
• credit assignment
• reinforcement learning