How Inference Compute Shapes Frontier LLM Evaluation

2026-06-16

AI Summary

Current evaluations of large language models commonly employ fixed computational budgets during inference, which inadequately capture their true capabilities on complex tasks. This work systematically investigates the impact of inference-time computational resources—including token budgets, context compression, and repeated submissions—on model performance across seven challenging benchmarks. Using a unified evaluation framework applied to multiple state-of-the-art models, the study reveals for the first time that the allocation of inference computation significantly influences assessment outcomes: larger token budgets consistently enhance performance across diverse domains, whereas fixed budgets systematically underestimate the capabilities of advanced models. The authors argue that model competence should be conceptualized as a function of inference-time computation and advocate for transparent reporting of evaluation protocols.

Abstract

AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving. As a result, performance is increasingly sensitive to the amount and allocation of compute available at test time ("inference compute"). Yet many evaluations still report performance at a single restrictive budget, meaning that low scores may reflect the evaluation setup rather than the model's underlying capability. To test this, we evaluate up to 12 frontier language models on seven challenging benchmarks spanning software engineering, mathematics, medicine, and cybersecurity. We use a controlled setup combining three simple inference-scaling interventions: larger token budgets, context compaction, and repeated submission attempts, guided either by the model itself or by minimal correctness feedback. We find three main results. First, larger token budgets substantially improve performance on benchmarks across multiple domains, including cybersecurity, FrontierMath, Humanity's Last Exam, and TerminalBench. Second, fixed-budget evaluations can increasingly understate frontier capability as models advance. Newer models reach higher performance at large budgets, where they unlock harder tasks and solve them more reliably. Third, benchmarks differ in which inference-scaling methods help most: repeated submission broadly improves performance, but the value of larger token budgets, external feedback, and parallel attempts varies by benchmark. Overall, our results show that benchmark scores are protocol-dependent. We therefore argue that evaluations should report capability as a function of inference-time compute, specify protocol choices explicitly, and compare model generations over a large shared compute range at matched budgets, especially in safety- or policy-relevant settings.

Problem

• inference compute
• LLM evaluation
• benchmarking
• token budget
• frontier models

Innovation

• inference compute
• evaluation protocol
• token budget
• repeated submission
• frontier LLMs