Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories

2026-07-12

AI Summary

This work addresses a key limitation of conventional supervised fine-tuning (SFT), which merely imitates expert action sequences without discerning the relative quality of actions at the state level. To overcome this, the authors propose a lightweight offline policy optimization method that transforms expert trajectories into state-conditioned single-step action preferences. Leveraging a DPO-style objective, the approach contrasts expert actions against sampled negative actions—eliminating the need for environment interactions or an explicit reward model. The method introduces Policy-Preserving Augmentation (PPA) to maintain policy consistency and, for the first time, enables state-level action preference learning from expert-only data in a fully offline setting. Experiments demonstrate substantial improvements over SFT, boosting the accuracy of a 9B model on tau-bench from 21.7% to 41.4%, matching the performance of online GRPO.

Abstract

Large Language Model (LLM) agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which treats multi-turn agent behavior as ordinary text imitation. This recipe is simple and low-cost, but it only learns to imitate the sequence of expert actions, rather than training the agent to choose the right action against plausible mistakes at each state. Existing methods to mitigate this problem include preference learning or reinforcement learning, but they usually need high-cost environment rollouts and reward models. We propose Agentic-DPO, a lightweight offline agent policy optimization method that turns expert trajectories into state-conditioned preference supervision. At each expert action state, Agentic-DPO samples a one-step action from the current state, treats plausible wrong actions as negatives, and contrasts them with the expert action using a DPO-style preference objective. To avoid mixing both policy and schema in preference learning, we introduce Policy-Preserving Augmentation (PPA), which renders the same latent trajectory under multiple schemas while keeping the expert policy fixed. Agentic-DPO requires no online environment rollout, reward model, or full-trajectory student exploration. We conduct experiments across StableToolBench, tau-bench retail, and Mind2Web, where Agentic-DPO consistently improves agents at different model scales beyond imitation. In particular, it raises tau-bench accuracy from 21.7% (SFT) to 41.4% for a 9B model, matching online GRPO under the same backbone with only step-level rollouts and without environment interaction during gradient steps. The results suggest that expert trajectories can support low-cost agentic policy optimization when converted from demonstrations into state-level action preferences. Code for Agentic-DPO is released at https://github.com/Schuture/Agentic-DPO.

Problem

• LLM agents
• expert trajectories
• imitation learning
• policy optimization
• preference learning

Innovation

• Agentic-DPO
• preference optimization
• expert trajectories
• policy-preserving augmentation
• offline reinforcement learning