The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning

2026-06-14

AI Summary

This work uncovers a "quality-utility paradox" in knowledge distillation for mathematical reasoning: high-quality data selected by strong models based on high reward scores often degrades the performance of smaller language models due to distributional shift. To address this, the authors propose a style-aligned refinement method that preserves the logical correction capabilities of the teacher model while maintaining the student model’s native reasoning trajectory, thereby mitigating distributional drift. The approach integrates knowledge distillation with rejection sampling and distribution alignment strategies. Extensive experiments across multiple small language models—including Qwen2.5, LLaMA-3, and DeepSeek—confirm the existence of the paradox and demonstrate that the proposed method not only recovers but also significantly enhances mathematical reasoning performance.

Abstract

Knowledge distillation from powerful reasoning models is widely used to improve Small Language Models (SLMs) on mathematical reasoning, often assuming that traces with higher reward model scores provide more useful supervision. We identify a counterintuitive \textbf{Quality-Utility Paradox} in mathematical reasoning distillation. Data refined or synthesized by a stronger Oracle obtains higher perceived quality according to reward models, yet consistently underperforms traces generated by the SLM itself and selected through rejection sampling across Qwen2.5, LLaMA-3, and DeepSeek families. Our analysis shows that Oracle refinement couples logical repair with distributional drift away from the SLM's native reasoning distribution. This drift increases the learner's adaptation cost and can outweigh the benefit of improved reasoning logic. To test this mechanism, we introduce \textbf{Style-Aligned Refinement}, which preserves the native trajectory of the SLM while retaining logical repair from the Oracle. This intervention lowers adaptation cost and restores downstream utility. These findings suggest that effective mathematical reasoning distillation should jointly optimize perceived solution quality and learner-data compatibility, rather than relying solely on reward-model scores. The datasets and code are available at https://github.com/Dracoqhl/Quality-Utility-Paradox.

Problem

• Quality-Utility Paradox
• mathematical reasoning
• knowledge distillation
• small language models
• reward models

Innovation

• Quality-Utility Paradox
• Mathematical Reasoning
• Knowledge Distillation
• Style-Aligned Refinement
• Distributional Drift