Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions

2026-08-08

AI Summary

This work addresses the limited ability of current reasoning language models to strategically allocate computational resources across heterogeneous problems under a shared token budget, particularly in optimizing solution order based on problem difficulty and point value. The authors propose an exam-style evaluation framework that requires models to solve multiple diverse problems within a fixed total token budget to maximize overall score, formally introducing the problem of global computational budget allocation for the first time. Through experiments combining explicit planning prompts with multitask reasoning setups on mathematical and coding tasks, they demonstrate that prevailing models predominantly adopt greedy sequential solving strategies. Even when augmented with planning prompts, these models fail to effectively prioritize problems according to their difficulty and associated rewards.

Abstract

Reasoning language models increasingly use test-time compute to improve performance, but existing evaluations typically study this compute one question at a time. Yet when multiple problems share an end-to-end cost or latency constraint, models must decide how to divide limited inference compute among them. We introduce an exam-style evaluation framework for studying this setting, in which a model must distribute one shared token budget across questions with different difficulty and point values to maximize its total score. Across several open and frontier reasoning models, we find that models fail to allocate a shared budget strategically across questions of varying difficulties and values. Models behave largely as greedy sequential solvers: they prioritize questions by presentation order, front-load effort on early questions, and remain insensitive to value, with these tendencies becoming more pronounced as the number of questions grows. Explicit planning prompts spread compute more evenly but do not produce value- or difficulty-aware prioritization. The same behavioral pattern extends from mathematical to code reasoning. These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation and remains a challenge for current reasoning models.

Problem

• test-time compute
• budget allocation
• reasoning models
• difficulty-aware prioritization
• shared token budget

Innovation

• test-time compute allocation
• reasoning models
• shared token budget
• global budget allocation
• exam-style evaluation