MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model

2025-10-13

AI Summary

Existing reinforcement learning (RL) methods for mathematical reasoning primarily refine pre-existing solution patterns of base models, failing to elicit genuinely novel reasoning capabilities; even with high sampling budgets (e.g., pass@1024), leading open-source models solve most problems in benchmarks like MATH-500 and AIME 2024. Method: We introduce MATH-Beyond—the first RL-specific benchmark explicitly designed to assess whether RL can transcend the inherent capability boundaries of base models. Its challenging problem set is curated from the hardest instances of DAPO-Math-17K and DeepScaleR, emphasizing generalization beyond sampling limits. Contribution/Results: Experiments show that state-of-the-art RL-finetuned models underperform base models significantly at pass@1024, confirming MATH-Beyond’s rigor and diagnostic value. This benchmark establishes a critical evaluation tool and opens new directions for exploration-driven RL research in mathematical reasoning.

Technology Category

• Machine Learning: Reinforcement Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

With the advent of DeepSeek-R1, a new wave of reinforcement learning (RL) methods has emerged that seem to unlock stronger mathematical reasoning. However, a closer look at the open-source ecosystem reveals a critical limitation: with sufficiently many draws (e.g., $ exttt{pass@1024}$), many existing base models already solve nearly all questions on widely used math benchmarks such as MATH-500 and AIME 2024. This suggests that the RL fine-tuning methods prevalent in the LLM reasoning literature largely sharpen existing solution modes rather than discovering entirely new ones. Such sharpening stands in contrast to the broader promise of RL: to foster exploration and to acquire new skills. To move beyond this plateau, we introduce MATH-Beyond (MATH-B), a benchmark deliberately constructed to defeat common open-source models of up to 8B parameters even under large sampling budgets. Improving performance on our benchmark via RL requires methods that learn to reason in ways that go beyond base model capabilities in repeated sampling. Since the problems are drawn from subsets of DAPO-Math-17K and DeepScaleR datasets, they remain topically equivalent to standard high-school math. Validating our premise, RL fine-tuned models such as Nemotron-Research-Reasoning-Qwen-1.5B and DeepScaleR-1.5B-Preview perform poorly on MATH-B at $ exttt{pass@1024}$, showing how existing approaches fall short on tackling harder instances. We hope MATH-B will catalyze exploration-driven RL approaches that elicit deeper reasoning capabilities. We release MATH-B at https://huggingface.co/datasets/brendel-group/MATH-Beyond.

Problem

• Existing math benchmarks fail to challenge base models with large sampling
• RL fine-tuning mainly sharpens existing skills rather than discovering new ones
• MATH-Beyond benchmark requires RL to develop novel reasoning capabilities

Innovation

• Introduced MATH-B benchmark to challenge base models
• Problems sourced from DAPO-Math-17K and DeepScaleR datasets
• Designed to require exploration-driven RL for deeper reasoning