COSEARCH: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search

2026-04-19

AI Summary

This work addresses a critical limitation in existing intelligent search methods, which treat the retrieval system as a fixed component and optimize only the reasoning agent, thereby making retrieval a performance bottleneck. To overcome this, the authors propose CoSearch, a novel framework that enables end-to-end joint training of multi-step reasoning agents and generative document ranking models for the first time. CoSearch introduces Group Relative Policy Optimization (GRPO), a policy gradient method based on semantic grouping, and designs a composite reward mechanism that integrates both ranking quality and feedback from reasoning trajectories. Evaluated across seven single-hop and multi-hop question answering benchmarks, CoSearch significantly outperforms strong baselines, demonstrating the effectiveness of joint training and the individual contributions of its core components.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Agentic search -- the task of training agents that iteratively reason, issue queries, and synthesize retrieved information to answer complex questions -- has achieved remarkable progress through reinforcement learning (RL). However, existing approaches such as Search-R1, treat the retrieval system as a fixed tool, optimizing only the reasoning agent while the retrieval component remains unchanged. A preliminary experiment reveals that the gap between an oracle and a fixed retrieval system reaches up to +26.8% relative F1 improvement across seven QA benchmarks, suggesting that the retrieval system is a key bottleneck in scaling agentic search performance. Motivated by this finding, we propose CoSearch, a framework that jointly trains a multi-step reasoning agent and a generative document ranking model via Group Relative Policy Optimization (GRPO). To enable effective GRPO training for the ranker -- whose inputs vary across reasoning trajectories -- we introduce a semantic grouping strategy that clusters sub-queries by token-level similarity, forming valid optimization groups without additional rollouts. We further design a composite reward combining ranking quality signals with trajectory-level outcome feedback, providing the ranker with both immediate and long-term learning signals. Experiments on seven single-hop and multi-hop QA benchmarks demonstrate consistent improvements over strong baselines, with ablation studies validating each design choice. Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant, pointing to a key ingredient for future search agents.

Problem

• agentic search
• retrieval system
• reasoning agent
• document ranking
• reinforcement learning

Innovation

• joint training
• reinforcement learning
• document ranking
• agentic search
• semantic grouping