Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization

2026-04-15

AI Summary

This work addresses the challenges of unstable value estimation and ineffective credit assignment in reinforcement learning–based LLM search agents under sparse trajectory-level rewards. To overcome these limitations, the authors propose the CW-GRPO framework, which innovatively incorporates an LLM-based judge to evaluate the quality of each retrieval and reasoning step, generating per-turn contribution scores. These scores are used to rescale trajectory advantages, enabling fine-grained credit assignment. By integrating such process supervision into Group Relative Policy Optimization (GRPO), CW-GRPO maintains training stability while significantly enhancing search performance. Experiments demonstrate that CW-GRPO outperforms standard GRPO across multiple knowledge-intensive benchmarks, achieving absolute gains of 5.0% and 6.3% with Qwen3-8B and Qwen3-1.7B, respectively, and revealing a consistent pattern of highly concentrated contributions within successful trajectories.

Technology Category

• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Search agents extend Large Language Models (LLMs) beyond static parametric knowledge by enabling access to up-to-date and long-tail information unavailable during pretraining. While reinforcement learning has been widely adopted for training such agents, existing approaches face key limitations: process supervision often suffers from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. To bridge this gap, we propose Contribution-Weighted GRPO (CW-GRPO), a framework that integrates process supervision into group relative policy optimization. Instead of directly optimizing process rewards, CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores. These scores are used to rescale outcome-based advantages along the trajectory, enabling fine-grained credit assignment without sacrificing optimization stability. Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0\% on Qwen3-8B and 6.3\% on Qwen3-1.7B, leading to more effective search behaviors. Additional analysis reveals that successful trajectories exhibit concentrated contributions across rounds, providing empirical insight into search agent tasks.

Problem

• search agents
• reinforcement learning
• credit assignment
• process supervision
• outcome supervision

Innovation

• Contribution-Weighted GRPO
• process supervision
• credit assignment
• search agents
• reinforcement learning