RL-Index: Reinforcement Learning for Retrieval Index Reasoning

2026-06-15

AI Summary

This work addresses the challenge of implicit and complex reasoning relationships between queries and knowledge, which traditional retrieval methods handle by query-side rewriting—resulting in high latency and an inability to leverage reasoning during indexing. To overcome this limitation, the paper formulates index construction as a reinforcement learning problem and introduces an index-side reasoning mechanism based on Group Relative Policy Optimization (GRPO). During indexing, large language models generate explicit rationales to augment documents, preemptively encoding latent query-knowledge relationships. The model is trained with retrieval similarity as a reward signal, enabling verifiable optimization. Evaluated on the BRIGHT benchmark, the approach significantly improves both retrieval effectiveness and downstream question answering performance, substantially reduces online latency, and demonstrates strong generalization and plug-and-play compatibility across diverse retrievers and generators.

Abstract

Retrieving external knowledge is essential for solving real-world tasks, yet it remains challenging when the relationship between a query and its relevant knowledge involves implicit and complex reasoning beyond surface-level semantic or lexical matching (e.g., mathematical problems relying on the same theorem or coding requiring deep reasoning). Existing approaches primarily rely on query-side reasoning (e.g., query rewriting), which introduces significant online latency and underutilizes the opportunity to perform reasoning over the knowledge corpus itself (i.e., index-side reasoning). In this paper, we propose RL-Index, an agentic indexing framework that formulates retrieval index reasoning as a reinforcement learning problem. Instead of performing reasoning at query time, RL-Index shifts reasoning to the indexing stage by augmenting documents with LLM-generated rationales that explicitly encode the latent query-knowledge relationship. To optimize the quality of these rationales, we employ Group Relative Policy Optimization (GRPO) and use retrieval similarity as a verifiable reward signal, enabling direct optimization of indexing decisions for retrieval effectiveness. Extensive experiments on the BRIGHT benchmark demonstrate that RL-Index consistently improves both retrieval and downstream question-answering performance, while significantly reducing online inference latency. Moreover, the learned rationale augmentation generalizes across diverse retrievers and generators, highlighting its robustness as a plug-and-play indexing strategy across different retrieval systems.

Problem

• retrieval
• reasoning
• indexing
• reinforcement learning
• knowledge retrieval

Innovation

• reinforcement learning
• index-side reasoning
• rationale augmentation
• retrieval optimization
• GRPO