Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval

2025-04-07

AI Summary

In generative information retrieval (GenIR), a misalignment exists between token-level document ID generation and document-level relevance estimation; existing reinforcement learning (RL)-based approaches rely on complex reward modeling and RL fine-tuning, suffering from poor training stability and high computational overhead. To address this, we propose Direct Document Relevance Optimization (DDRO), the first method to formulate document relevance optimization as a lightweight pairwise ranking problem—bypassing reward modeling and RL entirely. DDRO enables end-to-end joint optimization of token generation and document-level relevance discrimination within a unified generative framework, integrating sequence modeling, pairwise ranking loss, and relevance classification seamlessly. Evaluated on MS MARCO and Natural Questions, DDRO achieves +7.4% and +19.9% improvements in MRR@10 over state-of-the-art RL methods, respectively. It demonstrates superior effectiveness, generalizability, and training stability.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generative information retrieval (GenIR) is a promising neural retrieval paradigm that formulates document retrieval as a document identifier (docid) generation task, allowing for end-to-end optimization toward a unified global retrieval objective. However, existing GenIR models suffer from token-level misalignment, where models trained to predict the next token often fail to capture document-level relevance effectively. While reinforcement learning-based methods, such as reinforcement learning from relevance feedback (RLRF), aim to address this misalignment through reward modeling, they introduce significant complexity, requiring the optimization of an auxiliary reward function followed by reinforcement fine-tuning, which is computationally expensive and often unstable. To address these challenges, we propose direct document relevance optimization (DDRO), which aligns token-level docid generation with document-level relevance estimation through direct optimization via pairwise ranking, eliminating the need for explicit reward modeling and reinforcement learning. Experimental results on benchmark datasets, including MS MARCO document and Natural Questions, show that DDRO outperforms reinforcement learning-based methods, achieving a 7.4% improvement in MRR@10 for MS MARCO and a 19.9% improvement for Natural Questions. These findings highlight DDRO's potential to enhance retrieval effectiveness with a simplified optimization approach. By framing alignment as a direct optimization problem, DDRO simplifies the ranking optimization pipeline of GenIR models while offering a viable alternative to reinforcement learning-based methods.

Problem

• Align token-level docid generation with document-level relevance
• Simplify optimization without reward modeling or reinforcement learning
• Improve retrieval effectiveness in generative information retrieval models

Innovation

• Direct document relevance optimization via pairwise ranking
• Eliminates need for reward modeling and reinforcement learning
• Aligns token-level generation with document-level relevance