ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction

2026-08-12

AI Summary

This study addresses reward unfairness in reinforcement learning caused by behavioral diversity during open-ended interactions. We propose ARC, a method that restores fair comparison through policy-conditioned grouping, hybrid rewards, and entropy regularization, while integrating an INTER paradigm to decouple communication from reasoning for optimized agent interaction learning. Experimental results demonstrate significant performance improvements on tool-use benchmarks, with first-token response latency reduced from 4.91s to 1.27s. By effectively mitigating training bias and efficiency bottlenecks in complex interactive scenarios, this work establishes a novel paradigm for aligning open-domain agents.

Abstract

Open-ended real-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting. This flexibility breaks a core assumption behind group-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable. As a result, reward-model preferences over interaction style can distort relative advantages and steer optimization toward reward-preferred behaviors rather than context-appropriate ones. We formalize this as a \textit{reward fairness problem} and propose \textbf{ARC} (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison through strategy-conditioned rollout grouping, together with hybrid rewards and entropy regularization. We study ARC in our proposed \inter, a novel paradigm for responsive, steerable, and execution-aware user-agent interaction that decouples user-visible communication from latent reasoning and tool use. \inter\ also provides the annotation and distillation pipeline for constructing \inter-86K, our strategy-annotated training corpus for supervised and RL training. Empirically, ARC substantially strengthens the core $τ/τ^2$ tool-use benchmarks, while \inter\ reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline. Together, these results suggest that a central bottleneck in open-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place. The ARC implementation and \inter-86K training data will be released.

Problem

• Reward Fairness
• Open-Ended Interaction
• Relative Advantage
• Reinforcement Learning
• Behavioral Comparability

Innovation

• ARC
• Reward Fairness
• Strategy-Conditioned Rollout Grouping
• Inter Paradigm
• Open-Ended Interaction