Hit-RAG: Learning to Reason with Long Contexts via Preference Alignment

2026-03-07

AI Summary

This work addresses the challenge in long-context retrieval-augmented generation (RAG) where high information density often drowns critical evidence in noise, leading to attention dilution and reasoning hallucinations. To this end, we propose Hit-RAG, the first three-stage progressive preference alignment framework that integrates supervised fine-tuning (SFT), discriminative preference alignment (DPA), and group relative policy optimization (GRPO) to systematically enhance the model’s ability to identify salient evidence and perform accurate logical reasoning within lengthy contexts. Experimental results demonstrate that Hit-RAG significantly outperforms existing methods across eight benchmarks, effectively mitigating attention dilution and reasoning collapse. Notably, it enables smaller models to surpass larger counterparts on complex long-context reasoning tasks, thereby bridging the gap between context acquisition and precise reasoning.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Search and Optimization: Metareasoning and Metaheuristics
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite the promise of Retrieval-Augmented Generation in grounding Multimodal Large Language Models with external knowledge, the transition to extensive contexts often leads to significant attention dilution and reasoning hallucinations. The surge in information density causes critical evidence to be submerged by voluminous noise, which complicates the discernment of relevant fragments within a dense input. In this paper, we propose \textbf{Hit-RAG}, a multi-stage preference alignment framework designed to resolve these cognitive bottlenecks through a progressive optimization pipeline. Our approach systematically refines the utilization of external evidence via three distinct stages. First, Supervised Fine-tuning establishes baseline context awareness to minimize information neglect. Next, Discriminative Preference Alignment enhances robustness against misleading distractors. Finally, Group-Relative Policy Optimization stabilizes logical synthesis to prevent reasoning collapse. Extensive evaluations on eight benchmarks demonstrate that Hit-RAG consistently yields substantial performance gains, enabling models to bridge the gap between context acquisition and accurate reasoning while surpassing much larger counterparts in long-context scenarios.

Problem

• Retrieval-Augmented Generation
• attention dilution
• reasoning hallucinations
• long-context reasoning
• information noise

Innovation

• Retrieval-Augmented Generation
• Preference Alignment
• Long-context Reasoning
• Hallucination Mitigation
• Policy Optimization