Co-Evolving Policy Distillation

2026-04-29

AI Summary

This work addresses the performance degradation arising from behavioral discrepancies among experts during capability integration—such as policy divergence in hybrid RLVR and insufficient knowledge absorption in conventional online policy distillation (OPD). To mitigate this, we propose a co-evolutionary policy distillation framework that, for the first time, integrates bidirectional, synchronous online policy distillation into parallel reinforcement learning training. In this framework, expert models act as mutual teachers and students, co-evolving to maintain behavioral consistency while preserving complementary knowledge. By synergistically combining reinforcement learning with multimodal reasoning, our approach significantly outperforms strong baselines—including hybrid RLVR and MOPD—across text, image, and video tasks, and even surpasses single-domain expert models, thereby demonstrating its effectiveness and scalability.

Abstract

RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.

Problem

• Policy Distillation
• RLVR
• OPD
• Capability Consolidation
• Behavioral Pattern Gap

Innovation

• Co-Evolving Policy Distillation
• RLVR
• Online Policy Distillation
• Multi-expert Integration
• Behavioral Consistency