Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models

2026-08-17

AI Summary

This study addresses the unclear generalization mechanisms in policy distillation for large language models by employing controlled experiments and cross-domain evaluations to systematically reveal its dual nature. The findings demonstrate that distillation fundamentally transfers reasoning behaviors rather than merely replicating answers, while clarifying the differential impacts of homologous versus heterogeneous teachers and the trade-offs inherent in multi-teacher ensembles. Notably, this work establishes the boundary conditions for generalization in policy distillation and identifies a "seesaw" phenomenon in multi-teacher capabilities. These contributions provide critical theoretical support and novel perspectives for understanding knowledge transfer mechanisms in large models and for optimizing diagnostic approaches in multi-teacher distillation frameworks.

Abstract

On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher's influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosing multi-teacher OPD.

Problem

• On-Policy Distillation
• Generalization
• Large Language Models
• Multi-teacher Distillation
• Cross-domain Transfer

Innovation

• On-Policy Distillation
• Generalization
• Reasoning Behavior Transfer
• Multi-Teacher Distillation
• Same-Origin Relationship