Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation

2026-06-22

AI Summary

Existing policy distillation methods transfer only decision-making positions while neglecting the critical evidence tokens that underpin those decisions, resulting in incomplete reasoning knowledge transfer. This work proposes DEAR, a novel approach that explicitly distinguishes and jointly models two types of knowledge—decisions and evidence—at the token level. Decision points are identified via the student model’s prediction entropy, while evidence tokens are localized through a combination of hidden-state cosine similarity and representation discrepancies between teacher and student models. DEAR achieves substantial improvements over current policy distillation techniques on mathematical and code generation benchmarks, yielding gains of 2.5 and 5.7 percentage points on competitive mathematics and code generation tasks, respectively, thereby effectively addressing the gap in reasoning-aware knowledge distillation.

Abstract

On-policy distillation transfers reasoning ability through dense token-level supervision, yet the nature of the transferable signal remains unclear. We discover that reasoning chains contain two types of knowledge that require different discovery mechanisms: decisions (where to branch), which surface through student uncertainty, and evidence (intermediate steps that justify decisions), which hides in positions where the student is confident yet wrong. Current methods capture only decisions; the substantive knowledge in evidence tokens remains untransferred. We propose DEAR(Decision-Evidence Aware Reasoning Distillation), which first identifies decisions via student entropy, then discovers their supporting evidence through hidden-state cosine similarity to decision anchors, boosted by teacher-student divergence to prioritize the largest knowledge gaps. Across three student-teacher configurations on math and code benchmarks, DEAR consistently outperforms standard OPD, with up to +2.5pp on competition math and +5.7pp on code generation.

Problem

• on-policy distillation
• reasoning chains
• decision-supporting tokens
• evidence
• knowledge transfer

Innovation

• on-policy distillation
• reasoning chains
• decision-evidence discovery
• knowledge transfer
• cosine similarity