🤖 AI Summary
Large language models often suffer from suboptimal task performance due to inaccurate decisions about when to invoke external search. This work formulates search invocation as an instance-level routing problem and introduces three counterfactual oracles—no-search, search, and unsolvable—defined by task-specific success criteria to generate supervised signals for unified evaluation and training. Leveraging supervised fine-tuning and preference optimization guided by these success metrics, the proposed approach substantially improves routing accuracy. Experiments demonstrate that macro F1 scores for routing increase from 0.7082 to 0.8235 on Gemma E2B and from 0.7053 to 0.8365 on Qwen3.5-4B, effectively reducing both unnecessary and omitted search calls.
📝 Abstract
Search-augmented language models can use external evidence to compensate for limitations in parametric knowledge, but search is not uniformly beneficial: models may call search for questions they can already answer, or rely on noisy evidence when correction, clarification, or abstention would be more appropriate. We formulate this as an instance-level search-routing problem: deciding whether search is needed to improve task success relative to a no-search execution. To derive supervision, we compare no-search and forced-search outcomes for the same question and construct an oracle over NO SEARCH, SEARCH, and UNSOLVED based on task-specific success. Using this oracle as both an evaluation criterion and a learning signal, we train search-routing policies with supervised fine-tuning and preference optimization, improving routing macro-F1 on oracle-eligible examples from 0.7082 to 0.8235 for Gemma E2B and from 0.7053 to 0.8365 for Qwen3.5-4B. Further analysis shows that the learned policies reduce model-specific routing failures: Gemma primarily learns no-search restraint, while Qwen further reduces missed search; residual UNSOLVED cases reveal heterogeneous bottlenecks involving model capacity, retrieval budget, evidence use, and policy behavior.