Better RAG using Relevant Information Gain

2024-07-16

arXiv.org

AI Summary

To address the problem of critical information being displaced by redundant content in retrieval-augmented generation (RAG) systems due to limited context windows, this paper proposes a retrieval optimization method based on Relevance Information Gain (RIG). The method unifies relevance and diversity within a principled information-theoretic objective—eliminating the need for heuristic, explicit trade-off mechanisms—and enables diversity to emerge naturally in the optimal retrieval ranking. Leveraging probabilistic information-theoretic modeling and a plug-and-play retrieval re-ranking module, the approach integrates seamlessly with existing RAG pipelines. Evaluated on the RGB multi-task question-answering benchmark, our method significantly outperforms baselines including Maximal Marginal Relevance (MMR), achieving state-of-the-art performance. These results empirically validate the effectiveness and generalizability of an information-gain-driven retrieval paradigm for enhancing large language model (LLM) question-answering quality.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Learning to Search
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

A common way to extend the memory of large language models (LLMs) is by retrieval augmented generation (RAG), which inserts text retrieved from a larger memory into an LLM's context window. However, the context window is typically limited to several thousand tokens, which limits the number of retrieved passages that can inform a model's response. For this reason, it's important to avoid occupying context window space with redundant information by ensuring a degree of diversity among retrieved passages. At the same time, the information should also be relevant to the current task. Most prior methods that encourage diversity among retrieved results, such as Maximal Marginal Relevance (MMR), do so by incorporating an objective that explicitly trades off diversity and relevance. We propose a novel simple optimization metric based on relevant information gain, a probabilistic measure of the total information relevant to a query for a set of retrieved results. By optimizing this metric, diversity organically emerges from our system. When used as a drop-in replacement for the retrieval component of a RAG system, this method yields state-of-the-art performance on question answering tasks from the Retrieval Augmented Generation Benchmark (RGB), outperforming existing metrics that directly optimize for relevance and diversity.

Problem

• Optimizing retrieval for LLMs
• Enhancing diversity and relevance
• Improving RAG system performance

Innovation

• Relevant Information Gain metric
• Optimizes diversity and relevance organically
• Outperforms existing RAG retrieval methods