Problem
Research questions and friction points this paper is trying to address.
On-policy Distillation
Reasoning Progress
Teacher-derived Rewards
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
🤖 AI Summary
本文针对教师奖励与推理进展不匹配的问题，提出了一种基于推理进展感知的奖励过滤方法R2-OPD，以改善在策略蒸馏效果。
📝 Abstract
On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.
Innovation
Methods, ideas, or system contributions that make the work stand out.
On-Policy Distillation
Reasoning Progress
Reward Filtering
R2-OPD
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
