Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training

2026-07-02

AI Summary

This study investigates whether online policy self-distillation alone is sufficient to mitigate catastrophic forgetting and enable effective knowledge updating in continual post-training. Through the Self-Distillation Policy Optimization (SDPO) framework—augmented with analyses of parameter and representation space drift, as well as detection of high-frequency formatting artifacts—the authors find that while SDPO can accelerate in-domain specialization when teacher signals remain stable, dense self-distillation often induces significant representational drift. This drift, compounded by teacher-student feedback loops, amplifies formatting artifacts and degrades out-of-distribution generalization, sometimes leading to model collapse. In contrast, online reinforcement learning approaches such as GRPO adopt a more conservative update strategy, better preserving pre-existing capabilities. The work thus challenges the prevailing assumption that self-distillation inherently serves as a stable mechanism for continual learning, revealing its latent risks and delineating its operational boundaries.

Abstract

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.

Problem

• continual post-training
• on-policy self-distillation
• catastrophic forgetting
• distribution shift
• teacher-student loop

Innovation

• self-distillation
• continual post-training
• on-policy learning
• forgetting
• SDPO