MathDuels: Evaluating LLMs as Problem Posers and Solvers

2026-04-23

AI Summary

Current static mathematical benchmarks evaluate large language models solely as problem solvers, failing to distinguish performance differences among state-of-the-art models. This work proposes a self-play evaluation framework that jointly assesses a model’s ability to both generate and solve problems. The approach employs a three-stage adversarial prompting process to create questions, filters invalid items via an independent verifier, and leverages the Rasch model to co-estimate item difficulty and model proficiency. Experiments across 19 state-of-the-art models demonstrate that this method reveals capability gaps invisible to single-role benchmarks. Notably, newer models generate problems capable of defeating previously strong solvers, providing evidence for the partial decoupling of problem-generation and problem-solving abilities and highlighting the dynamic evolution of benchmark difficulty.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

As frontier language models attain near-ceiling performance on static mathematical benchmarks, existing evaluations are increasingly unable to differentiate model capabilities, largely because they cast models solely as solvers of fixed problem sets. We introduce MathDuels, a self-play benchmark in which models occupy dual roles: each authors math problems under adversarial prompting and solves problems authored by every other participant. Problems are produced through a three-stage generation pipeline (meta-prompting, problem generation, and difficulty amplification), and validated by an independent verifier that excludes ill-posed questions. A Rasch model (Rasch, 1993) jointly estimates solver abilities and problem difficulties; author quality is derived from the difficulties of each model's authored problems. Experiments across 19 frontier models reveal that authoring and solving capabilities are partially decoupled, and that dual-role evaluation reveals capability separations invisible in single-role benchmarks. As newer models enter the arena, they produce problems that defeat previously dominant solvers, so the benchmark's difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. We host a public leaderboard that updates as new models are released.

Problem

• mathematical reasoning
• language models
• benchmarking
• problem posing
• evaluation

Innovation

• self-play benchmark
• problem posing
• Rasch model
• difficulty co-evolution
• adversarial prompting