AMO-Bench: Large Language Models Still Struggle in High School Math Competitions

2025-10-30

AI Summary

Existing mathematical competition benchmarks suffer from performance saturation and data leakage, limiting their ability to discriminate among state-of-the-art large language models (LLMs) in rigorous mathematical reasoning. Method: We introduce AMO-Bench—the first original benchmark designed specifically for high-intensity mathematical reasoning evaluation—comprising 50 expert-cross-validated problems with difficulty at least on par with the International Mathematical Olympiad (IMO), supporting automated final-answer-only scoring. Our methodology integrates human-authored problem generation, multi-stage difficulty calibration, and scalability analysis. Contribution/Results: Evaluating 26 mainstream LLMs reveals a stark performance ceiling: the best-performing model achieves only 52.4% accuracy, while most score below 40%, exposing critical bottlenecks in complex deductive reasoning. Crucially, we demonstrate robust computational scalability—reasoning performance consistently improves with increased test-time compute resources—validating AMO-Bench’s utility for probing and advancing frontier reasoning capabilities.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present AMO-Bench, an Advanced Mathematical reasoning benchmark with Olympiad level or even higher difficulty, comprising 50 human-crafted problems. Existing benchmarks have widely leveraged high school math competitions for evaluating mathematical reasoning capabilities of large language models (LLMs). However, many existing math competitions are becoming less effective for assessing top-tier LLMs due to performance saturation (e.g., AIME24/25). To address this, AMO-Bench introduces more rigorous challenges by ensuring all 50 problems are (1) cross-validated by experts to meet at least the International Mathematical Olympiad (IMO) difficulty standards, and (2) entirely original problems to prevent potential performance leakages from data memorization. Moreover, each problem in AMO-Bench requires only a final answer rather than a proof, enabling automatic and robust grading for evaluation. Experimental results across 26 LLMs on AMO-Bench show that even the best-performing model achieves only 52.4% accuracy on AMO-Bench, with most LLMs scoring below 40%. Beyond these poor performances, our further analysis reveals a promising scaling trend with increasing test-time compute on AMO-Bench. These results highlight the significant room for improving the mathematical reasoning in current LLMs. We release AMO-Bench to facilitate further research into advancing the reasoning abilities of language models. https://amo-bench.github.io/

Problem

• Creating a benchmark with IMO-level math problems
• Evaluating LLMs' mathematical reasoning on challenging tasks
• Addressing performance saturation in existing math benchmarks

Innovation

• Created expert-validated IMO-level difficulty problems
• Used entirely original problems to prevent data leakage
• Implemented automatic grading with final-answer only format