GRITHopper: Decomposition-Free Multi-Hop Dense Retrieval

2025-03-10

AI Summary

Existing query-decomposition-based multi-hop retrieval methods break end-to-end differentiability and incur high computational overhead, while decomposition-free approaches underperform on long-range multi-hop reasoning and out-of-distribution generalization. This paper introduces GRITHopper-7B—the first decomposition-free, end-to-end differentiable generative-representational joint instruction-tuned dense retrieval model for multi-hop QA. Its key innovations are: (1) a unified training paradigm jointly optimizing causal language modeling and dense retrieval; and (2) a post-retrieval language modeling mechanism leveraging answer supervision to enhance context awareness and retrieval–generation alignment. GRITHopper-7B achieves state-of-the-art performance on both in-distribution and out-of-distribution multi-hop benchmarks, significantly improving long-range reasoning accuracy and cross-domain generalization. The model is publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Decomposition-based multi-hop retrieval methods rely on many autoregressive steps to break down complex queries, which breaks end-to-end differentiability and is computationally expensive. Decomposition-free methods tackle this, but current decomposition-free approaches struggle with longer multi-hop problems and generalization to out-of-distribution data. To address these challenges, we introduce GRITHopper-7B, a novel multi-hop dense retrieval model that achieves state-of-the-art performance on both in-distribution and out-of-distribution benchmarks. GRITHopper combines generative and representational instruction tuning by integrating causal language modeling with dense retrieval training. Through controlled studies, we find that incorporating additional context after the retrieval process, referred to as post-retrieval language modeling, enhances dense retrieval performance. By including elements such as final answers during training, the model learns to better contextualize and retrieve relevant information. GRITHopper-7B offers a robust, scalable, and generalizable solution for multi-hop dense retrieval, and we release it to the community for future research and applications requiring multi-hop reasoning and retrieval capabilities.

Problem

• Addresses limitations of decomposition-based multi-hop retrieval methods.
• Improves performance on long multi-hop and out-of-distribution tasks.
• Introduces GRITHopper-7B for scalable, generalizable dense retrieval.

Innovation

• Combines generative and representational instruction tuning
• Integrates causal language modeling with dense retrieval
• Enhances retrieval with post-retrieval language modeling