SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

2026-06-17

AI Summary

Standard multi-turn on-policy distillation suffers from performance limitations due to error accumulation in early stages, over-penalization of semantically plausible actions, and unreliable teacher supervision. This work proposes the first validator-free selective intervention framework, which dynamically determines whether to intervene on the student’s response at each interaction step by jointly leveraging environmental feedback and teacher judgments. The approach introduces teacher confidence weighting and loss normalization mechanisms to allocate supervisory signals on demand. Evaluated on ALFWorld, the method achieves a 13.3% absolute improvement in success rate on unseen scenarios over standard on-policy distillation. Ablation studies further confirm the complementary effectiveness of each proposed component.

Abstract

On-policy distillation (OPD) improves student models by training them on trajectories induced by their own policy, making it a promising approach for mitigating exposure bias in agent training. However, most OPD studies focus on single-turn settings, while realistic LLM agents interact with environments over multiple turns. In this regime, early errors can alter future observations and compound across the trajectory, and standard dense token-level OPD becomes brittle, as it may over-penalize semantically valid alternatives, reinforce local degeneracies such as repeated actions, and propagate unreliable teacher supervision on off-distribution histories. We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD. Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on. To further address compounding errors, SAGE-OPD weights token-level distillation by teacher confidence, reducing the influence of uncertain teacher distributions on corrupted or ambiguous histories. Finally, SAGE-OPD applies loss normalization to preserve the overall loss scale of standard OPD while retaining selective turn-level weighting. Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD. Ablation studies further demonstrate that turn-level intervention, teacher confidence weighting, and loss normalization provide complementary benefits. Our results suggest that effective multi-turn OPD should remain on-policy, but teacher supervision should be selectively allocated to turns where intervention is necessary and reliable.

Problem

• on-policy distillation
• multi-turn interaction
• exposure bias
• compounding errors
• teacher supervision

Innovation

• on-policy distillation
• multi-turn interaction
• selective intervention
• teacher confidence weighting
• loss normalization