Problem
Research questions and friction points this paper is trying to address.
working memory
large language models
interference
in-context superposition
memory limitations
Human-like Working Memory Interference in Large Language Models
🤖 AI Summary
This study investigates the mechanisms underlying human-like working memory capacity limitations in large language models (LLMs), even when full contextual access is available. Employing an N-back task paradigm alongside two-layer Transformer training, evaluations on pretrained LLMs, representational geometry analysis, interference suppression experiments, and attention probing, the work reveals for the first time that LLMs exhibit human-like working memory interference. The authors propose a “contextual superposition” mechanism, wherein recent memory items overlap in internal representations, constraining capacity; target information is recovered through cross-layer reorganization that suppresses interference. Results demonstrate that an LLM’s working memory performance correlates positively with its overall capability, replicates human memory biases, and that either disentangling competing memories or enhancing interference suppression significantly improves retrieval accuracy on target tasks.
📝 Abstract
Intelligent systems must maintain and manipulate task-relevant information online to adapt to dynamic environments and changing goals. This capacity, known as working memory, is fundamental to human reasoning and intelligence. Despite having on the order of 100 billion neurons, both biological and artificial systems exhibit limitations in working memory. This raises a key question: why do large language models (LLMs) show such limitations, given that transformers have full access to prior context through attention? We find that although a two-layer transformer can be trained to solve working memory tasks perfectly, a diverse set of pretrained LLMs continues to show working memory limitations. Notably, LLMs reproduce interference signatures observed in humans: performance degrades with increasing memory load and is biased by recency and stimulus statistics. Across models, stronger working memory capacity correlates with broader competence on standard benchmarks, mirroring its link to general intelligence in humans. Yet despite substantial variability in working memory performance, LLMs surprisingly converge on a common computational mechanism. Rather than directly copying the relevant memory item from context, models encode multiple memory items in entangled representations, such that successful recall depends on interference control -- actively suppressing task-irrelevant content to isolate the target for readout. Moreover, a targeted intervention that suppresses stimulus content information improves performance, providing causal support for representational interference. Together, these findings identify representational interference as a core constraint on working memory in pretrained LLMs, suggesting that working-memory limits in biological and artificial systems may reflect a shared computational challenge: selecting task-relevant information under interference.
Innovation
Methods, ideas, or system contributions that make the work stand out.
in-context superposition
working memory
interference
large language models
representation overlap
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
No related jobs found.
