Adaptive Test-Time Compute Allocation with Evolving In-Context Demonstrations

2026-04-22

AI Summary

Existing test-time compute allocation methods often rely on static strategies or fixed generation distributions, struggling to balance performance and efficiency. This work proposes the first adaptive framework that jointly optimizes compute allocation and generation strategy. It employs a two-stage mechanism: during a warm-up phase, it identifies easy queries and constructs an initial example pool; in the adaptive phase, it dynamically updates contextual prompts based on semantic similarity, directing computational resources toward challenging samples. The approach substantially outperforms current methods across mathematical, programming, and reasoning benchmarks while significantly reducing inference compute overhead.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While scaling test-time compute can substantially improve model performance, existing approaches either rely on static compute allocation or sample from fixed generation distributions. In this work, we introduce a test-time compute allocation framework that jointly adapts where computation is spent and how generation is performed. Our method begins with a warm-up phase that identifies easy queries and assembles an initial pool of question-response pairs from the test set itself. An adaptive phase then concentrates further computation on unresolved queries while reshaping their generation distributions through evolving in-context demonstrations -- conditioning each generation on successful responses from semantically related queries rather than resampling from a fixed distribution. Experiments across math, coding, and reasoning benchmarks demonstrate that our approach consistently outperforms existing baselines while consuming substantially less inference-time compute.

Problem

• test-time compute allocation
• adaptive computation
• in-context learning
• generation distribution
• query difficulty

Innovation

• test-time compute allocation
• in-context learning
• adaptive generation
• evolving demonstrations
• compute-efficient inference