Reasoning Quality Emerges Early: Data Curation for Reasoning Models

2026-06-25

AI Summary

Existing approaches to filtering high-quality reasoning data rely heavily on strong reasoning models, resulting in high costs and limited effectiveness. This work proposes an efficient alternative that reliably identifies challenging samples by analyzing the loss of a pretrained model over the first 100 reasoning tokens. By further incorporating loss patterns and gradient similarity from a small number of perturbed checkpoints, the method enables precise selection of diverse, high-difficulty data without requiring complex inference procedures. Evaluated on Qwen2.5-7B and Llama3.1-8B, this approach achieves up to a 1.7% improvement in fine-tuning performance while reducing token consumption by 91%, substantially lowering the cost of data curation.

Abstract

Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs). However, existing methods for curating high-quality SFT data rely heavily on strong reasoning models to filter examples based on diversity and difficulty, making the curation process costly while often yielding suboptimal data quality. In this work, we show that diverse and challenging reasoning examples can be identified using only the initial reasoning tokens. Specifically, we demonstrate that difficult problems can be reliably detected based on the loss of the first 100 reasoning tokens evaluated at a randomly perturbed checkpoint of the pretrained model. We further show that examples exhibiting similar loss patterns over their first 1k reasoning tokens across a small number of perturbed checkpoints extrapolating along the fine-tuning trajectory provably induce similar gradients. We validate our approach through extensive experiments on fine-tuning Qwen2.5-7B and Llama3.1-8B models on the M23K medical reasoning and OpenThoughts-Math datasets. Our method outperforms existing baselines by up to 1.7% while being 91% more token efficient.

Problem

• reasoning quality
• data curation
• supervised fine-tuning
• Large Language Models
• reasoning traces

Innovation

• reasoning data curation
• early token loss
• gradient similarity
• supervised fine-tuning
• token efficiency