A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning

2025-07-10

AI Summary

Existing methods lack a systematic design for the synergistic interplay between supervised fine-tuning (SFT) and reinforcement learning (RL) in mathematical reasoning. Method: We propose a two-stage training paradigm: first, extending the SFT phase to substantially improve reasoning accuracy; second, applying the GRPO algorithm—trained on online inference trajectories—to optimize generation efficiency by compressing solution paths. Contribution/Results: We are the first to explicitly characterize the complementary roles of SFT (focusing on accuracy breakthroughs) and GRPO (specializing in path compression). Our approach achieves top-0.5% performance (ranking among the best in over 2,200 teams) on authoritative benchmarks including the AI Math Olympiad, significantly outperforming single-stage baselines. All code and models are publicly released to ensure full reproducibility.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Enhancing the mathematical reasoning of Large Language Models (LLMs) is a pivotal challenge in advancing AI capabilities. While Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are the dominant training paradigms, a systematic methodology for combining them to maximize both accuracy and efficiency remains largely unexplored. This paper introduces a practical and effective training recipe that strategically integrates extended SFT with RL from online inference (GRPO). We posit that these methods play complementary, not competing, roles: a prolonged SFT phase first pushes the model's accuracy to its limits, after which a GRPO phase dramatically improves token efficiency while preserving this peak performance. Our experiments reveal that extending SFT for as many as 10 epochs is crucial for performance breakthroughs, and that the primary role of GRPO in this framework is to optimize solution length. The efficacy of our recipe is rigorously validated through top-tier performance on challenging benchmarks, including a high rank among over 2,200 teams in the strictly leak-free AI Mathematical Olympiad (AIMO). This work provides the community with a battle-tested blueprint for developing state-of-the-art mathematical reasoners that are both exceptionally accurate and practically efficient. To ensure full reproducibility and empower future research, we will open-source our entire framework, including all code, model checkpoints, and training configurations at https://github.com/analokmaus/kaggle-aimo2-fast-math-r1.

Problem

• Enhancing mathematical reasoning in LLMs with SFT and RL
• Maximizing accuracy and efficiency in AI mathematical problem-solving
• Developing a reproducible framework for state-of-the-art mathematical reasoners

Innovation

• Extended SFT maximizes model accuracy
• GRPO enhances token efficiency post-SFT
• Combined SFT and RL optimize performance