PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

2026-06-15

AI Summary

This work addresses key challenges in post-training multi-turn tool-using agents, including sparse rewards, weak credit assignment, and the overly restrictive nature of expert trajectory supervision. The authors propose PACT, a novel framework that treats expert trajectories as privileged signals exclusively during training, rather than as prompts required at inference time. PACT integrates trajectory-conditioned reinforcement learning with component-aware supervised fine-tuning and introduces a prompt anchoring mechanism to reduce sensitivity to trajectory context. A unified optimization objective is achieved through latent trajectory modeling. Evaluated on FTRL, BFCL, and ToolHop benchmarks, PACT significantly outperforms strong supervised fine-tuning and reinforcement learning baselines, demonstrating the efficacy of co-training with privileged trajectory information.

Abstract

Multi-turn tool-use agents must reason, call tools, and adapt to observations across several interaction turns. Post-training such agents is challenging, as reinforcement learning often suffers from sparse rewards and weak credit assignment despite matching the prompt-only inference setting, while supervised fine-tuning on expert traces provides dense process supervision but can over-constrain the model to fixed trajectories. To tackle this, we propose PACT, a Privileged trAce Co-Training framework for multi-turn tool-use agents. The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints. PACT keeps rollout generation prompt-only, then uses expert traces to guide optimization through two complementary signals: a trace-conditioned RL surrogate that evaluates prompt-only rollouts under expert-trace context, and a component-aware SFT loss that supervises reasoning prefixes and tool-calls with annealed strength. To reduce over-reliance on the training-only trace context, PACT further introduces a prompt-only anchoring. We also provide a latent-trace view that connects the two trace-based objectives and explains how expert traces can guide optimization without being used during rollout generation. Experiments on FTRL, BFCL, and ToolHop show that PACT consistently improves over strong SFT- and RL-based baselines, highlighting the value of privileged trace co-training for multi-turn tool-use learning.

Problem

• multi-turn tool-use agents
• post-training
• sparse rewards
• expert traces
• over-constrained trajectories

Innovation

• privileged trace co-training
• multi-turn tool-use agents
• trace-conditioned RL
• component-aware SFT
• prompt-only anchoring