HeteRAG: A Heterogeneous Retrieval-augmented Generation Framework with Decoupled Knowledge Representations

2025-04-12

AI Summary

Existing RAG methods employ homogeneous knowledge chunk representations across retrieval and generation stages, forcing retrieval to rely on lengthy text chunks for coverage while exposing generation to redundant information—degrading both output quality and inference efficiency. To address this, we propose the first heterogeneous representation framework tailored for RAG: the retrieval stage leverages multi-granular, context-augmented long chunks to maximize recall, whereas the generation stage operates on concise, semantically cohesive short chunks to enhance fidelity and speed. We further introduce an adaptive prompt tuning mechanism that jointly optimizes both pathways. Evaluated on multiple benchmarks, our approach consistently outperforms state-of-the-art RAG methods, achieving superior generation quality (e.g., +3.2 BLEU, +4.1 ROUGE-L) and up to 38% lower latency—demonstrating a principled trade-off between effectiveness and efficiency.

Technology Category

• Natural Language Processing: Generation
• Knowledge Representation and Reasoning: Argumentation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-augmented generation (RAG) methods can enhance the performance of LLMs by incorporating retrieved knowledge chunks into the generation process. In general, the retrieval and generation steps usually have different requirements for these knowledge chunks. The retrieval step benefits from comprehensive information to improve retrieval accuracy, whereas excessively long chunks may introduce redundant contextual information, thereby diminishing both the effectiveness and efficiency of the generation process. However, existing RAG methods typically employ identical representations of knowledge chunks for both retrieval and generation, resulting in suboptimal performance. In this paper, we propose a heterogeneous RAG framework (myname) that decouples the representations of knowledge chunks for retrieval and generation, thereby enhancing the LLMs in both effectiveness and efficiency. Specifically, we utilize short chunks to represent knowledge to adapt the generation step and utilize the corresponding chunk with its contextual information from multi-granular views to enhance retrieval accuracy. We further introduce an adaptive prompt tuning method for the retrieval model to adapt the heterogeneous retrieval augmented generation process. Extensive experiments demonstrate that myname achieves significant improvements compared to baselines.

Problem

• Decouples knowledge representations for retrieval and generation
• Improves retrieval accuracy with multi-granular contextual information
• Enhances generation efficiency by avoiding redundant contextual data

Innovation

• Decouples knowledge representations for retrieval and generation
• Uses multi-granular views to enhance retrieval accuracy
• Adaptive prompt tuning for heterogeneous RAG process