Does Engram Do Memory Retrieval in Autoregressive Image Generation?

2026-05-13

AI Summary

This study investigates whether the Engram module in autoregressive image generation relies on a content-addressable memory mechanism. We adapt Engram to the visual domain by introducing 2D spatial n-gram hashing, gated fusion, and KV-cache-compatible incremental inference, integrating it into an ImageNet 256×256 class-conditional generator for systematic analysis. Experimental results show that all Engram-enhanced models exhibit worse FID than the baseline; a minimal fixed gate performs comparably to a learned gate; and freezing a randomly initialized memory table degrades FID by only 0.10 while improving the Inception Score. These findings suggest that Engram does not function as a content-addressable memory but rather as a gated bypass mechanism, challenging its original interpretation in language models.

Abstract

The Engram module -- a hash-keyed, O(1) associative memory injected into Transformer layers -- was recently shown to improve large language model pretraining, with the appealing interpretation that it provides a content-addressed shortcut to recurring local token patterns. We ask whether this interpretation transfers to autoregressive (AR) image generation, or whether the observed gains, if any, come from a different mechanism. We adapt the Engram module to vision with 2D spatial $n$-gram hashing, gated fusion, and KV-cache-compatible incremental inference, and inject it into a class-conditional AR generator trained on ImageNet 256x256. Across a sweep of backbone-to-memory budget ratios $ρ{\in}[0.17, 0.90]$, every Engram-augmented variant trails the pure AR baseline in FID, indicating that the module saves backbone FLOPs but does not, by itself, improve sample quality. We then probe how the module is used. A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic, yet a tiny constant gate (g=0.10) matches or beats the learned gate -- inconsistent with a heavily content-addressed recall mechanism. A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions, while collapsing or randomising the table degrades them by two to three orders of magnitude. Finally, training a model from scratch with the entire memory table frozen to $\mathcal{N}(0, 1)$ noise costs only $Δ\text{FID}{=}0.10$ and actually raises Inception Score. Together, these findings indicate that the Engram in AR image generation behaves not as a content-addressed retriever but as a gated architectural side-pathway: a hash-keyed residual stream whose benefit is dominated by the pathway itself, with the learned table contributing only a small distributional refinement.

Problem

• Engram
• autoregressive image generation
• content-addressed memory
• memory retrieval
• Transformer

Innovation

• Engram module
• autoregressive image generation
• content-addressed memory
• 2D spatial n-gram hashing
• gated residual pathway