A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications

2025-10-19

AI Summary

Large language models (LLMs) suffer from static knowledge bases, factual hallucinations, and difficulty accessing real-time or domain-specific information; conventional retrieval-augmented generation (RAG) typically employs single-step, heuristic retrieval without dynamic control over retrieval and reasoning. To address these limitations, this work introduces a reinforcement learning (RL)-driven agent search paradigm. We propose an adaptive search architecture integrating LLMs, RAG, multi-step interaction, and RL—enabling autonomous planning, iterative retrieval, and reflective revision. Methodologically, we establish the first unified theoretical framework for RL-based search agents, structured along three dimensions: functional roles, optimization methodologies, and application scopes. We further survey state-of-the-art approaches, standardized evaluation protocols, and open-source resources, and delineate concrete pathways toward scalable and reliable agent development. This work provides a systematic foundation and practical guidance for advancing dynamic, knowledge-enhanced intelligent agents. (149 words)

Technology Category

• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The advent of large language models (LLMs) has transformed information access and reasoning through open-ended natural language interaction. However, LLMs remain limited by static knowledge, factual hallucinations, and the inability to retrieve real-time or domain-specific information. Retrieval-Augmented Generation (RAG) mitigates these issues by grounding model outputs in external evidence, but traditional RAG pipelines are often single turn and heuristic, lacking adaptive control over retrieval and reasoning. Recent advances in agentic search address these limitations by enabling LLMs to plan, retrieve, and reflect through multi-step interaction with search environments. Within this paradigm, reinforcement learning (RL) offers a powerful mechanism for adaptive and self-improving search behavior. This survey provides the first comprehensive overview of emph{RL-based agentic search}, organizing the emerging field along three complementary dimensions: (i) What RL is for (functional roles), (ii) How RL is used (optimization strategies), and (iii) Where RL is applied (scope of optimization). We summarize representative methods, evaluation protocols, and applications, and discuss open challenges and future directions toward building reliable and scalable RL driven agentic search systems. We hope this survey will inspire future research on the integration of RL and agentic search. Our repository is available at https://github.com/ventr1c/Awesome-RL-based-Agentic-Search-Papers.

Problem

• Addresses limitations of static LLMs and heuristic RAG systems
• Explores reinforcement learning for adaptive multi-step agentic search
• Surveys RL roles, optimizations, and applications in search systems

Innovation

• Reinforcement learning optimizes agentic search behavior
• Multi-step interaction enables planning and reflection
• Adaptive control over retrieval and reasoning processes