Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation

2025-01-24

AI Summary

Existing mathematical reasoning benchmarks suffer from small scale, low quality, severe evaluation contamination, and unreliable results. Method: We design an automated data pipeline to mine Olympiad-level problems and community solutions from the Art of Problem Solving (AoPS) forum, enabling scalable, low-cost generation of expert-level mathematical instruction data; we further propose a time-aware, multi-dimensional evaluation framework. Contribution/Results: We release AoPS-Instruct—a high-quality, 600K+ QA dataset—and LiveAoPSBench, a contamination-immune, timestamped dynamic benchmark grounded in real-time forum data. Fine-tuned models achieve significant gains on mainstream mathematical reasoning benchmarks. LiveAoPSBench provides the first empirical evidence that LLM performance decays markedly with problem publication time—strongly supporting the pretraining contamination hypothesis. All data and code are publicly available.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Metareasoning and Metaheuristics
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Advances in Large Language Models (LLMs) have sparked interest in their ability to solve Olympiad-level math problems. However, the training and evaluation of these models are constrained by the limited size and quality of available datasets, as creating large-scale data for such advanced problems requires extensive effort from human experts. In addition, current benchmarks are prone to contamination, leading to unreliable evaluations. In this paper, we present an automated pipeline that leverages the rich resources of the Art of Problem Solving (AoPS) forum, which predominantly features Olympiad-level problems and community-driven solutions. Using open-source LLMs, we develop a method to extract question-answer pairs from the forum, resulting in AoPS-Instruct, a dataset of more than 600,000 high-quality QA pairs. Our experiments demonstrate that fine-tuning LLMs on AoPS-Instruct improves their reasoning abilities across various benchmarks. Moreover, we build an automatic pipeline that introduces LiveAoPSBench, an evolving evaluation set with timestamps, derived from the latest forum data, providing a contamination-resistant benchmark for assessing LLM performance. Notably, we observe a significant decline in LLM performance over time, suggesting their success on older examples may stem from pre-training exposure rather than true reasoning ability. Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning, offering valuable insights into the capabilities and limitations of LLMs in this domain. Our benchmark and code is available at https://github.com/DSL-Lab/aops

Problem

• Large Language Models
• Complex Mathematical Problems
• Evaluation Accuracy

Innovation

• Automated Training Methodology
• AoPS-Instruct Dataset
• LiveAoPSBench Evaluation