EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models

2026-07-12

AI Summary

Existing online policy distillation (OPD) methods suffer from reproducibility and extensibility challenges due to inconsistencies in supervision formulations, tokenizer compatibility, and implementation details. To address this, we propose EasyOPD—the first modular and extensible unified framework for OPD—built upon the verl distributed reinforcement learning platform. EasyOPD decouples user configuration, method-specific logic, and execution backends, offering standardized interfaces for critical components such as loss construction, metadata handling, and tokenizer alignment. The framework features YAML-driven configuration, dynamic rollout-based supervision collection, and encapsulated teacher computation, enabling plug-and-play support for diverse OPD approaches. We successfully reproduce three representative OPD methods across four benchmarks—inference, code generation, scientific knowledge, and tool use—while preserving their original performance characteristics. Code and configurations are publicly released.

Abstract

Conventional language-model distillation often relies on fixed teacher-generated data, which may not cover the states encountered by an evolving student policy. On-policy distillation (OPD) instead collects teacher or evaluator supervision on student-generated rollouts. However, existing OPD methods differ substantially in supervision form, tokenizer compatibility, teacher access, and supervision granularity, leading to fragmented implementations that are difficult to reproduce and extend. We present \textsc{EasyOPD}, an on-policy distillation framework built on verl, a distributed reinforcement-learning framework for large language models. \textsc{EasyOPD} separates user-side configuration, method-specific supervision logic, and verl-based execution. Its method modules connect to the shared backend through extension boundaries for loss construction, rollout metadata, reward processing, tokenizer alignment, and teacher-side computation. We instantiate representative methods for three OPD settings -- cross-tokenizer OPD, on-policy self-distillation, and step-wise OPD. Experiments on reasoning, code-generation, scientific-knowledge, and tool-use benchmarks show that these implementations can be executed through the same verl-based backend while retaining their method-specific objectives and task-dependent performance profiles. We release \textsc{EasyOPD} with runnable YAML configurations, documentation, and an installable demonstration package and video.

Problem

• on-policy distillation
• language model distillation
• fragmented implementations
• tokenizer compatibility
• supervision granularity

Innovation

• On-Policy Distillation
• Large Language Models
• Framework Modularization
• Tokenizer Alignment
• Reinforcement Learning