Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad

2025-03-27

AI Summary

Existing mathematical evaluation benchmarks assess only final answers, neglecting rigorous reasoning and formal proof generation capabilities. Method: This work presents the first systematic evaluation of large language models (LLMs) on the complete 2025 USAMO—six proof-based problems—emphasizing constructive proof synthesis over answer matching. We introduce a fine-grained, competition-oriented scoring framework integrating expert human annotation, stepwise reasoning trajectory analysis, and failure-mode attribution, applied to state-of-the-art reasoning models including o3-mini. Contribution/Results: All evaluated models achieve an average score below 5%, exposing fundamental deficiencies—including logical gaps, proof hallucinations, and structural incoherence—demonstrating that current LLMs remain incapable of performing high-fidelity, formally rigorous reasoning required for advanced mathematical problem solving.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent math benchmarks for large language models (LLMs) such as MathArena indicate that state-of-the-art reasoning models achieve impressive performance on mathematical competitions like AIME, with the leading model, o3-mini, achieving scores comparable to top human competitors. However, these benchmarks evaluate models solely based on final numerical answers, neglecting rigorous reasoning and proof generation which are essential for real-world mathematical tasks. To address this, we introduce the first comprehensive evaluation of full-solution reasoning for challenging mathematical problems. Using expert human annotators, we evaluated several state-of-the-art reasoning models on the six problems from the 2025 USAMO within hours of their release. Our results reveal that all tested models struggled significantly, achieving less than 5% on average. Through detailed analysis of reasoning traces, we identify the most common failure modes and find several unwanted artifacts arising from the optimization strategies employed during model training. Overall, our results suggest that current LLMs are inadequate for rigorous mathematical reasoning tasks, highlighting the need for substantial improvements in reasoning and proof generation capabilities.

Problem

• Evaluating LLMs on rigorous math proof generation
• Assessing failure modes in reasoning for USAMO problems
• Identifying training artifacts affecting mathematical reasoning

Innovation

• Evaluating full-solution reasoning for math problems
• Using expert human annotators for model assessment
• Identifying failure modes in model reasoning traces