SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval

2026-04-08

AI Summary

This work addresses the challenge that complex queries require multi-step reasoning, yet existing large language models struggle to reliably generate high-quality reasoning paths in unsupervised settings. The paper proposes SubSearch, a novel framework that introduces an intrinsic process-based reward mechanism—requiring neither external annotations nor auxiliary reward models—to directly optimize the generator via reinforcement learning. This approach guides the model to autonomously plan robust reasoning trajectories without supervision. By integrating multi-hop question answering with retrieval-augmented generation, SubSearch substantially outperforms baselines that rely solely on outcome-based rewards across seven benchmark datasets. The method not only enhances performance on complex question answering but also establishes a data-efficient paradigm for self-directed reasoning.

Abstract

Large language models (LLMs) are probabilistic in nature and perform more reliably when augmented with external information. As complex queries often require multi-step reasoning over the retrieved information, with no clear or predetermined reasoning path, they remain challenging. Recent approaches train models using reinforcement learning on the model's outcome, showing promise in improving how models handle complex information. We introduce SubSearch, a specialized framework that shifts from outcome-only supervision to intermediate reward signals that incentivize planning high-quality reasoning. Unlike previous work on process reward modeling, which focuses on training a separate reward model with annotated trajectories by either human annotators or large LLM judges, SubSearch directly optimizes the generator using intrinsic process rewards, which we define as internally-derived rewards, eliminating the need for external supervision, and moving towards autonomous information-intensive reasoning. Experiments on seven benchmarks show that rewarding intermediate reasoning steps with intrinsic rewards leads to more robust reasoning traces in both QA and multi-hop QA datasets over using only outcome rewards. SubSearch can help in building reasoning traces that allow agents to better integrate search engines for complex query answering, while offering a data-efficient alternative to supervised process modeling.

Problem

• complex retrieval
• multi-step reasoning
• intermediate rewards
• unsupervised reasoning
• information-intensive QA

Innovation

• intermediate rewards
• unsupervised reasoning
• intrinsic process rewards
• SubSearch
• multi-hop QA