Trust Region Policy Distillation

2026-07-06

AI Summary

This work addresses the instability in online policy distillation (OPD), which often stems from high gradient variance. To mitigate this issue, the authors propose a trust-region-based policy distillation framework that dynamically constructs a proximal teacher policy, thereby recasting OPD into a stable training paradigm with intrinsically controlled gradient variance. By integrating insights from proximal policy optimization, the method provides, for the first time, a global convergence guarantee and a monotonic improvement bound on policy performance—all without incurring additional computational overhead. Empirical evaluations demonstrate that the proposed approach significantly enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks.

Abstract

Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable, high-variance On-Policy Distillation (OPD) into a stable training paradigm by dynamically constructing a proximal teacher. Theoretically, we establish a rigorous framework demonstrating that TOP-D inherently controls gradient variance. By providing a formal global convergence analysis alongside a monotonic improvement bound, we mathematically formalize the reliability and stability of the overall training dynamics. Empirically, TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. More importantly, TOP-D introduces zero additional computational overhead, positioning itself as a promising alternative to the well-established OPD paradigm.

Problem

• On-Policy Distillation
• training instability
• gradient variance
• policy distillation
• trust region

Innovation

• Trust Region Policy Distillation
• On-Policy Distillation
• gradient variance control
• global convergence
• sample efficiency