One Policy, Any Budget: Internalizing Budget-Aware Search via Reinforcement Learning

2026-09-01

AI Summary

本文提出AnySearch框架，通过强化学习使单一策略在任意预算约束下执行搜索任务，解决了现有方法无法适应变化约束的问题。

Abstract

While reinforcement learning has enabled LLM-based search agents to invoke external tools, existing methods train under fixed budgets and cannot adapt when constraints vary at deployment. We propose AnySearch, a framework that enables a single policy to perform budget-aware search under any budget constraint through a training scaffold and curriculum reinforcement learning. In the first phase, we train the agent with explicit budget state injection and structured reasoning prompts that guide efficient allocation under linearly decaying budgets. In the second phase, the scaffold is removed and the agent learns to operate autonomously under adaptively sampled budget constraints, matching inference conditions. Both phases are optimized with a composite reward that couples answer accuracy with budget efficiency through absolute and relative signals, where an adaptive weight amplifies the efficiency signal for high-accuracy queries and attenuates it for low-accuracy ones. Extensive experiments on seven general and multi-hop QA benchmarks show that our method outperforms baselines across all budget scales, generalizes to unseen constraints beyond the training range, and achieves superior tool productivity without excessive token overhead. Our code is available at https://github.com/xwsun01/AnySearch.

Problem

• reinforcement learning
• budget-aware search
• adaptability
• fixed budget
• deployment constraints

Innovation

• budget-aware search
• curriculum reinforcement learning
• composite reward
• adaptive weight