What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code

2026-05-19

AI Summary

This study investigates the genuine role of code in enhancing mathematical reasoning, revealing that executable code itself is not the critical factor; rather, cross-domain structured reasoning traces—such as code-text and math-text hybrids—are central. Through fine-grained domain-separated pretraining experiments on a 10T-token corpus, combined with expert activation routing analysis and controlled modulation of structured mathematical sample density, the authors find that training exclusively on code actually impairs complex mathematical reasoning. In contrast, augmenting structured mathematical data substantially improves mathematical capabilities without compromising programming performance. The work proposes a cognition-scaffolded data optimization strategy that effectively mitigates inter-domain competition and empirically validates the existence of a cross-domain synergistic mechanism.

Abstract

Code has become a standard component of modern foundation language model (LM) training, yet its role beyond programming remains unclear. We revisit the claim that code improves reasoning through controlled pretraining experiments on a 10T-token corpus with fine-grained domain separation. Our findings are threefold. First, when code is restricted to standalone executable programs and Code-NL data are controlled for, code substantially improves programming ability but does not act as a general reasoning enhancer; instead, it competes with knowledge-intensive tasks, especially complex mathematical reasoning. Second, the reasoning gains often attributed to code are better explained by cross-domain structured reasoning traces, such as code-text and math-text mixtures, rather than by executable code alone. Third, increasing the density of structured math-domain samples within a fixed math budget yields substantial gains on difficult mathematical reasoning while largely preserving programming performance, suggesting that cognitive scaffolds offer a targeted way to mitigate cross-domain trade-offs. Finally, routing analyses show that data-composition effects are reflected in expert-activation patterns, providing mechanism-level evidence for competitive and synergistic interactions across domains. Our results clarify which data characteristics transfer across capability dimensions and point to more precise data-centric optimization strategies.

Problem

• mathematical reasoning
• code
• structured reasoning
• cross-domain transfer
• data composition

Innovation

• structured reasoning
• code-data trade-offs
• mathematical reasoning
• expert activation
• data-centric optimization