MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

2026-06-29

AI Summary

This work addresses the challenge of efficiently integrating multi-domain capabilities into large language models during post-training, where existing approaches often suffer from low efficiency or performance degradation. The authors propose a Multi-teacher Online Policy Distillation (MOPD) framework that first trains multiple domain-specialized teacher models via reinforcement learning and then performs exposure-bias-free online distillation using self-generated data from a student model. MOPD enables the first exposure-bias-free parallel distillation from multiple teachers, decouples cross-domain dependencies, and facilitates independent development of teacher models. Experiments on Qwen3-30B-A3B demonstrate that MOPD significantly outperforms baselines such as Mix-RL and Cascade RL, nearly preserving the full capabilities of individual teachers, and has been successfully deployed in the industrial-scale post-training of MiMo-V2-Flash.

Abstract

Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.

Problem

• capability integration
• large language models
• post-training
• multi-domain learning
• reinforcement learning

Innovation

• Multi-Teacher Distillation
• On-Policy Learning
• Capability Integration
• Reinforcement Learning Post-Training
• Exposure Bias Elimination