Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agents

2026-03-13

AI Summary

This work addresses the inefficiency of large language model (LLM) agents in multi-hop reasoning, where uncontrolled computation often leads to redundant steps and existing approaches struggle to intervene dynamically during inference. The authors propose Budget-Aware Value Trees (BAVT), a framework that models reasoning as a dynamic search tree and employs a single LLM to estimate step-level values. BAVT dynamically adjusts node selection based on the proportion of remaining computational budget, incorporating a training-free budget-conditioned selection mechanism and a residual value predictor. This design enables a smooth transition from exploration to exploitation, mitigates overconfidence in self-evaluation, and provides theoretical convergence guarantees under limited budgets. Experiments demonstrate that BAVT significantly outperforms parallel sampling baselines across four multi-hop question answering benchmarks, achieving superior performance even under low-budget settings compared to brute-force search methods using four times more resources.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Planning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Test-time scaling has become a dominant paradigm for improving LLM agent reliability, yet current approaches treat compute as an abundant resource, allowing agents to exhaust token and tool budgets on redundant steps or dead-end trajectories. Existing budget-aware methods either require expensive fine-tuning or rely on coarse, trajectory-level heuristics that cannot intervene mid-execution. We propose the Budget-Aware Value Tree (BAVT), a training-free inference-time framework that models multi-hop reasoning as a dynamic search tree guided by step-level value estimation within a single LLM backbone. Another key innovation is a budget-conditioned node selection mechanism that uses the remaining resource ratio as a natural scaling exponent over node values, providing a principled, parameter-free transition from broad exploration to greedy exploitation as the budget depletes. To combat the well-known overconfidence of LLM self-evaluation, BAVT employs a residual value predictor that scores relative progress rather than absolute state quality, enabling reliable pruning of uninformative or redundant tool calls. We further provide a theoretical convergence guarantee, proving that BAVT reaches a terminal answer with probability at least $1-ε$ under an explicit finite budget bound. Extensive evaluations on four multi-hop QA benchmarks across two model families demonstrate that BAVT consistently outperforms parallel sampling baselines. Most notably, BAVT under strict low-budget constraints surpasses baseline performance at $4\times$ the resource allocation, establishing that intelligent budget management fundamentally outperforms brute-force compute scaling.

Problem

• budget-aware reasoning
• LLM agents
• resource efficiency
• multi-hop reasoning
• compute budget

Innovation

• Budget-Aware Reasoning
• Value Tree Search
• Test-Time Scaling
• Residual Value Prediction
• Resource-Constrained LLM Agents