MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval

2026-04-20

AI Summary

Existing benchmarks for mathematical reasoning and retrieval suffer from limitations in scale, language coverage, and task diversity, hindering comprehensive evaluation of large language models’ mathematical capabilities. This work proposes MathNet, a large-scale multimodal and multilingual dataset encompassing 20 years of International Mathematical Olympiad problems from 47 countries across 17 languages. It further introduces the first human-annotated retrieval benchmark specifically designed to assess mathematical equivalence and structural similarity. The study establishes a unified evaluation framework centered on three core tasks: problem solving, math-aware retrieval, and retrieval-augmented problem solving. Experimental results show that leading models such as Gemini-3.1-Pro and GPT-5 achieve accuracies of 78.4% and 69.3%, respectively, while retrieval augmentation improves performance by up to 12%, with DeepSeek-V3.2-Speciale demonstrating the strongest overall results.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity. We introduce MathNet, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad-level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mathematical retrieval in embedding-based systems. MathNet spans 47 countries, 17 languages, and two decades of competitions, comprising 30,676 expert-authored problems with solutions across diverse domains. In addition to the core dataset, we construct a retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts. MathNet supports three tasks: (i) Problem Solving, (ii) Math-Aware Retrieval, and (iii) Retrieval-Augmented Problem Solving. Experimental results show that even state-of-the-art reasoning models (78.4% for Gemini-3.1-Pro and 69.3% for GPT-5) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that retrieval-augmented generation performance is highly sensitive to retrieval quality; for example, DeepSeek-V3.2-Speciale achieves gains of up to 12%, obtaining the highest scores on the benchmark. MathNet provides the largest high-quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark at https://mathnet.mit.edu.

Problem

• mathematical reasoning
• multimodal benchmark
• mathematical retrieval
• large language models
• Olympiad-level problems

Innovation

• Mathematical Reasoning
• Multimodal Benchmark
• Math-Aware Retrieval
• Retrieval-Augmented Generation
• Olympiad-Level Dataset