Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback

2026-06-12

AI Summary

This work addresses the challenge of balancing domain-specific expertise and cross-domain synergistic improvement in multi-domain large language model training. The authors propose a bidirectional collaborative distillation framework (OPCoD), wherein models excelling in distinct domains serve as reciprocal online mentors through policy-based feedback, enabling equitable and continuous co-evolution. To ensure the effectiveness and timeliness of knowledge transfer during distillation, the framework incorporates a cognitive gating mechanism and a feedback anchoring strategy, further enhanced by conditional self-distillation. Experimental results demonstrate that OPCoD significantly outperforms baseline methods on tasks such as Science Q&A, achieving Pareto improvements across all evaluated domain combinations and student architectures—marking the first realization of truly bidirectional collaborative model optimization.

Abstract

We study multi-domain LLM training in which two models, each stronger in a different domain, co-evolve by tutoring each other through on-policy feedback. Unlike one-way distillation or single-model fine-tuning, our goal is mutual Pareto improvement: each model improves across domains without losing its original strength. To this end, we propose On-Policy Co-Distillation (OPCoD), where each student's self-distillation is conditioned on its own correct rollout and feedback from its peer. To make feedback exchange effective, OPCoD uses cognizance-based gating to decide when to give feedback and feedback anchoring to ground feedback in the problem. On Science Q\&A tasks, OPCoD consistently outperforms baselines and achieves Pareto improvement across all evaluated domain pairs and students.

Problem

• multi-domain LLM training
• mutual improvement
• peer feedback
• Pareto improvement
• co-distillation

Innovation

• On-Policy Co-Distillation
• Mutual LLM Improvement
• Peer Feedback
• Cognizance-Based Gating
• Pareto Improvement