Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling

2026-08-12

AI Summary

This study investigates whether on-policy distillation (OPD) genuinely extends the reasoning capabilities of student models or merely improves sampling efficiency. Through a test-time scaling lens, the authors systematically evaluate OPD’s performance across varying sampling budgets \( K \) using both pass@K and avg@K metrics, complemented by problem-level solvability analysis benchmarked against pass@1024. The findings reveal that while OPD enhances avg@K under small \( K \), it is consistently outperformed by the original baseline in pass@K at large \( K \). Moreover, OPD renders some originally solvable problems unsolvable. These results suggest that OPD functions primarily as a form of “hallucination distillation,” optimizing sampling efficiency without substantively expanding the boundaries of reasoning capability.

Abstract

On-policy distillation (OPD) has emerged as a promising post-training technique for enhancing LLM reasoning. It is commonly believed to enable the student model to distill knowledge from a stronger teacher model, thereby expanding capabilities beyond the pre-OPD base model. In this study, we examine this view through the lens of test-time scaling by varying the sampling budget K and evaluating performance with pass@K and avg@K. Specifically, across several OPD variants, we observe that OPD-trained models maintain superior avg@K performance across sampling budgets, while the advantage in pass@K gradually shifts to the pre-OPD base models as K increases. These results suggest that OPD primarily improves sampling efficiency rather than consistently expanding the student's reasoning capability boundary. The pass@K dynamics throughout OPD training further reveal a progressive shift toward stronger small-K performance at the expense of the large-K capability boundary. Furthermore, a problem-level solvability analysis using pass@1024 as the criterion reveals an asymmetry: OPD causes more previously solvable problems to become unsolvable than previously unsolvable problems to become solvable. Together, these findings suggest that, from the perspective of capability expansion, OPD behaves more like an "illusory distillation": its apparent gains arise primarily from improved sampling efficiency rather than from acquiring genuinely new reasoning capabilities from the teacher.

Problem

• on-policy distillation
• reasoning capability
• test-time scaling
• sampling efficiency
• capability expansion

Innovation

• on-policy distillation
• test-time scaling
• sampling efficiency
• reasoning capability boundary
• pass@K