Human vs Machine Mathematical Difficulty on Project Euler: An Experimental Analysis

2026-06-20

AI Summary

This study investigates how the computational cost and success rate of state-of-the-art AI systems in solving Project Euler mathematical problems scale with human-perceived difficulty. Leveraging the MathArena benchmark, the authors analyze 3,840 attempts across 50 problems and 26 model configurations, providing the first empirical evidence that AI solution cost follows a sublinear power-law relationship with human solving time (exponent b < 1), while the log-transformed success rate exhibits a strong linear correlation with human time (median R² = 0.92). The work introduces a novel metric—50% task-length horizon (h₅₀)—to quantify AI mathematical proficiency, revealing that current state-of-the-art models achieve h₅₀ values of 2.5–4.3 hours, with capability approximately doubling every 75 days.

Abstract

We study how the effort and success probability of frontier AI systems scale with human difficulty on problems from Project Euler, an online platform of computational mathematics problems. Our dataset, from the MathArena benchmark, consists of 3840 attempts across 50 problems and 26 model configurations, with problem difficulty measured by the site's public human solve times. Motivated by a proposal of Timothy Gowers, we test a power-law relation $t_{\text{machine}} = a \cdot t_{\text{human}}^b$ between generated-token cost per successful answer and human time, and find $b < 1$ for 20 of the 25 models with usable fits, including the strongest base models; this operationalization therefore does not support an earlier prediction that machines scale worse than humans with difficulty. We also investigate whether success probability on the tested problems can be modeled by a simple exponential decay $p_{\text{success}} = e^{c t_{\text{human}}}$, predicting a linear relation between $\log p_{\text{success}}$ and $t_{\text{human}}$. Using a binning approach for data aggregation we find moderate empirical support (median bin-level $R^2 = 0.92$ across the 22 best-covered configurations) for this model. Following METR, we also fit logistic success curves and extract 50\% task-length horizons $h_{50}$; the strongest configurations in our 20 April 2026 snapshot reach roughly $2.5$--$4.3$ hours on our fastest-five human baseline, with a log-linear fit through the state-of-the-art frontier giving a descriptive doubling time of about $75$~days for the SOTA $h_{50}$.

Problem

• AI scaling
• mathematical problem solving
• human difficulty
• success probability
• computational effort

Innovation

• scaling law
• mathematical reasoning
• success probability modeling
• Project Euler
• task-length horizon