Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents

2025-05-17

AI Summary

LLM-based search agents suffer from severe efficiency bottlenecks in dynamic problem decomposition and alternating reasoning-retrieval execution: precise retrieval incurs high latency, while coarse-grained retrieval necessitates additional reasoning; inefficient scheduling and retrieval-induced blocking cause cascading delays, where even minor retrieval latencies are significantly amplified. Method: We propose SearchAgent-X—the first high-throughput, low-latency inference framework for LLM search agents supporting priority-aware scheduling and non-blocking retrieval. Contributions/Results: Its core innovations include high-recall approximate retrieval, dynamic priority-driven task scheduling, decoupled integration of retrieval and reasoning, and end-to-end LLM inference optimization. Experiments demonstrate that, without any degradation in generation quality, SearchAgent-X achieves 3.4× higher throughput and 5× lower end-to-end latency compared to state-of-the-art systems including vLLM and HNSW.

Technology Category

• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Model (LLM)-based search agents have shown remarkable capabilities in solving complex tasks by dynamically decomposing problems and addressing them through interleaved reasoning and retrieval. However, this interleaved paradigm introduces substantial efficiency bottlenecks. First, we observe that both highly accurate and overly approximate retrieval methods degrade system efficiency: exact search incurs significant retrieval overhead, while coarse retrieval requires additional reasoning steps during generation. Second, we identify inefficiencies in system design, including improper scheduling and frequent retrieval stalls, which lead to cascading latency -- where even minor delays in retrieval amplify end-to-end inference time. To address these challenges, we introduce SearchAgent-X, a high-efficiency inference framework for LLM-based search agents. SearchAgent-X leverages high-recall approximate retrieval and incorporates two key techniques: priority-aware scheduling and non-stall retrieval. Extensive experiments demonstrate that SearchAgent-X consistently outperforms state-of-the-art systems such as vLLM and HNSW-based retrieval across diverse tasks, achieving up to 3.4$ imes$ higher throughput and 5$ imes$ lower latency, without compromising generation quality. SearchAgent-X is available at https://github.com/tiannuo-yang/SearchAgent-X.

Problem

• Efficiency bottlenecks in LLM-based search agents
• Ineffective retrieval methods degrading system performance
• System design inefficiencies causing cascading latency

Innovation

• High-recall approximate retrieval for efficiency
• Priority-aware scheduling to reduce delays
• Non-stall retrieval to prevent latency amplification