Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Process

2026-06-19

AI Summary

The mechanisms by which large language models extract specific attribute knowledge from entity representations remain poorly understood. This work introduces the concept of “attribute computation paths” and employs an iterative layer ablation approach to identify the minimal subset of layers necessary for factual retrieval in LLaMA-3.1 8B and Qwen3 8B models. The study reveals that knowledge retrieval relies on multiple non-contiguous, functionally equivalent distributed pathways that can bypass intermediate layers, demonstrating a high degree of redundancy and strong distributedness in how large models compute knowledge. These findings challenge prevailing linear assumptions about knowledge storage and editing in transformer-based architectures.

Abstract

Large language models (LLMs) store and recall factual knowledge, yet the precise mechanism of how entity representations are transformed to enable specific attribute retrieval remains underexplored. In this work, we investigate this mechanism through the lens of an "attribute-computation path"-a sequence of computational steps over the entity representation required to elicit a target attribute. We then propose an iterative patching protocol to identify a minimal subset of layers necessary for this computation. Applying our method to LLaMA 3.1 8B and Qwen3 8B, we find that these paths are non-contiguous, often skipping layers, and that models possess multiple, functionally-equivalent paths for the same entity and fact, highlighting a high degree of redundancy in attribute computation. This implies that knowledge computation is highly distributed, potentially explaining the localization-editing mismatch and suggesting that knowledge storage and retrieval in LLMs is far from being well understood.

Problem

• factual retrieval
• large language models
• attribute computation
• knowledge representation
• redundancy

Innovation

• attribute-computation path
• iterative patching
• non-contiguous layers
• redundant retrieval
• distributed knowledge