How Retrieved Context Shapes Internal Representations in RAG

2026-02-23

AI Summary

This study investigates how the relevance and utility of retrieved documents in Retrieval-Augmented Generation (RAG) influence the internal representations and generation behavior of large language models (LLMs). Through controlled experiments, hidden state analyses, and cross-dataset, multi-model comparisons, the authors systematically evaluate representation shifts across model layers under single- and multi-document settings on four question-answering benchmarks and three LLMs. They provide the first internal-representation-level evidence that the relevance of retrieval context and its interaction with layer depth critically shape the model’s information integration mechanisms. The findings demonstrate a strong correlation between representational changes and generation quality, offering a theoretical foundation for understanding RAG output behavior and informing the design of more effective RAG systems.

Technology Category

• Machine Learning: Representation Learning
• Knowledge Representation and Reasoning: Argumentation
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by conditioning generation on retrieved external documents, but the effect of retrieved context is often non-trivial. In realistic retrieval settings, the retrieved document set often contains a mixture of documents that vary in relevance and usefulness. While prior work has largely examined these phenomena through output behavior, little is known about how retrieved context shapes the internal representations that mediate information integration in RAG. In this work, we study RAG through the lens of latent representations. We systematically analyze how different types of retrieved documents affect the hidden states of LLMs, and how these internal representation shifts relate to downstream generation behavior. Across four question-answering datasets and three LLMs, we analyze internal representations under controlled single- and multi-document settings. Our results reveal how context relevancy and layer-wise processing influence internal representations, providing explanations on LLMs output behaviors and insights for RAG system design.

Problem

• retrieval-augmented generation
• internal representations
• retrieved context
• large language models
• context relevancy

Innovation

• retrieval-augmented generation
• latent representations
• internal representations
• context relevance
• large language models