SSRL: Self-Search Reinforcement Learning

2025-08-14

AI Summary

This work addresses the over-reliance on external search engines in reinforcement learning (RL). We propose Self-Search RL, the first framework to treat large language models (LLMs) as trainable, autonomous search simulators. Methodologically, we explicitly model the search process via structured prompting, repeated sampling, and fine-grained reward shaping based on output format and domain-specific rules—thereby mitigating hallucination and enabling end-to-end self-iterative optimization. Our core contribution is endowing LLMs with intrinsic, RL-trainable search capabilities, eliminating the need for external search APIs. Experiments demonstrate that the trained policy significantly improves pass@k on open-domain question answering, drastically reduces dependency on external search, and exhibits strong cross-task transferability and promising generalization from simulation to real-world deployment.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We investigate the potential of large language models (LLMs) to serve as efficient simulators for agentic search tasks in reinforcement learning (RL), thereby reducing dependence on costly interactions with external search engines. To this end, we first quantify the intrinsic search capability of LLMs via structured prompting and repeated sampling, which we term Self-Search. Our results reveal that LLMs exhibit strong scaling behavior with respect to the inference budget, achieving high pass@k on question-answering benchmarks, including the challenging BrowseComp task. Building on these observations, we introduce Self-Search RL (SSRL), which enhances LLMs' Self-Search capability through format-based and rule-based rewards. SSRL enables models to iteratively refine their knowledge utilization internally, without requiring access to external tools. Empirical evaluations demonstrate that SSRL-trained policy models provide a cost-effective and stable environment for search-driven RL training, reducing reliance on external search engines and facilitating robust sim-to-real transfer. We draw the following conclusions: 1) LLMs possess world knowledge that can be effectively elicited to achieve high performance; 2) SSRL demonstrates the potential of leveraging internal knowledge to reduce hallucination; 3) SSRL-trained models integrate seamlessly with external search engines without additional effort. Our findings highlight the potential of LLMs to support more scalable RL agent training.

Problem

• Reducing reliance on costly external search engines
• Enhancing LLMs' internal knowledge utilization via reinforcement learning
• Creating cost-effective simulation for search-driven RL training

Innovation

• Self-Search RL with structured prompting and sampling
• Format-based and rule-based rewards enhance knowledge utilization
• Internal knowledge refinement reduces external search dependency