Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions

2025-05-26

AI Summary

The key data attributes driving mathematical and code reasoning capabilities in large language models (LLMs) remain poorly understood. Method: We propose a fine-grained, influence-function-based attribution framework enabling sample-level, sequence-level, and token-level interpretability. Contribution/Results: We identify for the first time cross-domain influence patterns between mathematical and coding tasks; introduce a “difficulty inversion” data reweighting strategy; and reveal divergent sequence-level exploration behaviors versus token-level logical/syntactic preferences. Evaluated on Qwen2.5-7B-Instruct, our approach improves AIME24 accuracy by 10 percentage points (10% → 20%) and LiveCodeBench score by 1.5 points (33.8% → 35.3%). Crucially, we demonstrate that high-difficulty mathematical samples exert positive transfer effects on both tasks—establishing difficulty as a critical, task-agnostic data property for enhancing LLM reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have demonstrated remarkable reasoning capabilities in math and coding, often bolstered by post-training on the chain-of-thoughts (CoTs) generated by stronger models. However, existing strategies for curating such training data predominantly rely on heuristics, limiting generalizability and failing to capture subtleties underlying in data. To address these limitations, we leverage influence functions to systematically attribute LLMs' reasoning ability on math and coding to individual training examples, sequences, and tokens, enabling deeper insights into effective data characteristics. Our Influence-based Reasoning Attribution (Infra) uncovers nontrivial cross-domain effects across math and coding tasks: high-difficulty math examples improve both math and code reasoning, while low-difficulty code tasks most effectively benefit code reasoning. Based on these findings, we introduce a simple yet effective dataset reweighting strategy by flipping task difficulty, which doubles AIME24 accuracy from 10% to 20% and boosts LiveCodeBench accuracy from 33.8% to 35.3% for Qwen2.5-7B-Instruct. Moreover, our fine-grained attribution reveals that the sequence-level exploratory behaviors enhance reasoning performance in both math and code, and the token-level influence patterns are distinct for math and code reasoning: the former prefers natural language logic connectors and the latter emphasizes structural syntax.

Problem

• Identify data attributes enhancing math and code reasoning in LLMs
• Investigate cross-domain effects of math and coding task difficulty
• Develop influence-based dataset reweighting to improve reasoning accuracy

Innovation

• Uses influence functions for data attribution
• Introduces dataset reweighting by flipping difficulty
• Reveals token-level influence patterns differences