Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL

2026-08-14

AI Summary

This study addresses the misalignment between teacher trustworthiness and policy objectives in agent self-distillation by proposing Influence-Calibrated Self-Distillation. Leveraging first-order influence functions to measure token-level gradient responses to reinforcement learning objectives, this method adaptively assigns supervision weights, shifting teacher guidance from trust-oriented to utility-oriented without additional inference overhead. Experiments demonstrate that the approach achieves a 96.1% success rate on ALFWorld and a score of 93.1 on WebShop, while reducing ineffective supervision by 37.8% and improving gradient compatibility by 0.192. These results indicate significant enhancements in both decision-making performance and training stability for language agents.

Abstract

On-policy self-distillation (OPSD) gives language agents dense token-level supervision from a privileged self-teacher on the policy's own trajectories. Existing methods allocate this supervision mainly by teacher trust, but trust does not reveal whether emphasizing a token supports the current policy objective. We call this the trust-utility mismatch and introduce Influence Calibration for Self-Distillation (ICSD). For each supervised token, ICSD measures the first-order response of its importance-weighted RL surrogate contribution to a teacher-directed output perturbation. Batch-adaptive calibration converts this non-stationary signal into a bounded allocation weight while preserving the original auxiliary-loss mass within each action turn. These detached weights affect only the distillation loss and require no additional model pass. Across ALFWorld, WebShop, and Search-QA, ICSD improves all matched aggregate metrics over trust-only allocation under Group Relative Policy Optimization (GRPO) and Group-in-Group Policy Optimization (GiGPO), across two model families spanning 1.5B to 7B. At 7B, it reaches 96.1% ALFWorld success and a WebShop score of 93.1. Frozen-batch analyses show that ICSD reduces teacher-supported mass assigned to objective-opposed tokens from 60.1% to 37.8% and raises cosine compatibility with the RL gradient by 0.192. A companion repository is avail- able at https://github.com/lanqz7766/Influence-Calibration-for-On-Policy-Self-Distillation-in-Agentic-RL.

Problem

• On-policy self-distillation
• Trust-utility mismatch
• Agentic RL
• Supervision allocation

Innovation

• Influence Calibration
• On-Policy Self-Distillation
• Trust-Utility Mismatch
• Agentic RL
• Gradient Compatibility