AIMO Interpretability Challenge

2026-07-15

AI Summary

Current reasoning benchmarks struggle to distinguish robust reasoning from spurious correlations in mathematical language models. This work addresses this gap by integrating interpretability and generalization research within the context of mathematical reasoning, leveraging problems from the AI Mathematical Olympiad (AIMO) and resources from the Fields Model Initiative. It introduces the first open robustness benchmark specifically designed for olympiad-level mathematical reasoning. By employing symbolic problem representations, adversarial robustness evaluations, functional variant generation, and interpretability analyses, the project delivers state-of-the-art models, a newly released dataset, and baseline systems. These resources collectively enable the identification of genuine reasoning mechanisms in models and advance the development of rigorous evaluation standards for reliable AI decision-making.

Abstract

We propose the AIMO Interpretability Challenge, a competition on distinguishing robust from spurious reasoning in frontier mathematical language models based on the models' internal mechanisms. The challenge is motivated by a central limitation of standard reasoning benchmarks: strong final-answer accuracy does not reveal whether a model relies on stable reasoning mechanisms or exploits brittle reasoning shortcuts. Building on AI Mathematical Olympiad (AIMO) problems and submissions, together with resources from the Fields Model Initiative, the competition will provide (1) newly-published olympiad-level math reasoning problems and their symbolic representations, allowing generation of novel functional variants, (2) access to frontier reasoning models, and (3) assessments of models' adversarial robustness on these problems. Participants will use these resources, along with our computing infrastructure support, to develop methods for identifying which models solve problems robustly. Our competition will also create a new, open robustness benchmark and baseline systems, aiming to provide a lasting foundation for standard benchmarking in mathematical reasoning and interpretability. Scientifically, the competition connects interpretability and generalization research around a central question in AI research: can we determine if, and to what extent, the decision-making of frontier AI models is generalizable and thus, reliable?

Problem

• interpretability
• mathematical reasoning
• robustness
• generalization
• spurious reasoning

Innovation

• interpretability
• mathematical reasoning
• robustness
• frontier language models
• adversarial evaluation