Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation

2026-04-14

AI Summary

This work addresses the high computational cost of standard online policy distillation (OPD), which relies on real-time teacher model inference, and the performance degradation of existing offline methods due to teacher inconsistency. The authors propose Lightning OPD, which formalizes for the first time the notion of “teacher consistency” and proves its critical role in converging to an optimal policy. By precomputing the teacher’s log-probabilities over trajectories, Lightning OPD enables fully offline, teacher-consistent distillation during supervised fine-tuning—eliminating the need for online teacher serving. Combining gradient bias analysis with implicit regularization, the method achieves 69.9% accuracy on AIME 2024 using only 30 GPU-hours on Qwen3-8B-Base, yielding a 4× speedup over standard OPD and substantially reducing post-training costs for large language models.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, standard OPD requires a live teacher inference server throughout training, resulting in substantial infrastructure overhead. In this work, we investigate whether on-policy distillation can be performed offline. A natural approach is to precompute teacher log-probabilities once over SFT rollouts and reuse them during training. In practice, however, this offline variant fails to reliably match the performance of standard OPD. To understand this discrepancy, we identify a previously overlooked condition that is critical for any OPD pipeline, which we term teacher consistency. This condition requires that the same teacher model be used for both supervised fine-tuning and OPD. We show that violating teacher consistency introduces an irreducible gradient bias, causing both offline and online OPD to converge to a suboptimal fixed point regardless of training duration. Building on this insight, we propose Lightning OPD, an offline on-policy distillation framework that enforces teacher consistency by precomputing teacher log-probabilities over SFT rollouts. This design eliminates the need for a live teacher server entirely. We further show that, under teacher consistency, Lightning OPD shares the same optimum as standard OPD, with bounded gradient discrepancy and an implicit regularization effect that helps prevent policy drift. Extensive experiments on mathematical reasoning and code generation demonstrate that Lightning OPD achieves state-of-the-art performance with significantly improved efficiency. Starting from an SFT-initialized Qwen3-8B-Base model, Lightning OPD reaches 69.9% on AIME 2024 in just 30 GPU hours, achieving a 4.0x speedup over standard OPD and substantially lowering the barrier to entry for academic research on LLM post-training.

Problem

• on-policy distillation
• post-training
• teacher consistency
• offline distillation
• large language models

Innovation

• offline on-policy distillation
• teacher consistency
• post-training efficiency
• large reasoning models
• gradient bias