Rubric-based On-policy Distillation

2026-05-08

AI Summary

This work addresses the limitation of conventional logits-based on-policy distillation, which is restricted to white-box settings and inapplicable to black-box teacher models. The authors propose ROPD, a novel framework that introduces structured rubrics into on-policy distillation for the first time. By generating prompt-specific rubrics through comparison of teacher and student outputs, ROPD scores student rollouts using only the teacher’s textual responses—without requiring access to teacher logits—and optimizes the student policy via on-policy reinforcement learning. This approach enables efficient black-box knowledge transfer, outperforming state-of-the-art logit-based methods across most tasks and achieving up to a 10× improvement in sample efficiency. ROPD thus establishes a general and scalable paradigm for distilling both closed- and open-source large language models.

Abstract

On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official.

Problem

• on-policy distillation
• black-box alignment
• teacher logits
• rubric-based distillation
• model alignment

Innovation

• rubric-based distillation
• on-policy distillation
• black-box alignment
• sample efficiency
• LLM distillation