Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers

2026-08-31

AI Summary

为解决视觉变换器中信息组织问题，提出分层赫布记忆架构，通过工作记忆、情景记忆和语义记忆三级结构及控制器进行有效信息管理。

Abstract

Vision Transformers provide strong visual representations but typically rely on slowly updated parameters, limiting their ability to organize newly acquired information across different memory timescales. This work proposes \textit{Hierarchical Hebbian Memory}, a three-level memory architecture composed of rapid Working Memory, persistent Routed Episodic Memory, and slower Semantic Memory. A learned controller regulates memory contribution, read and write routing, plasticity, retention, and consolidation. A causal read-before-write lifecycle ensures that the current outcome cannot influence the prediction it supervises. The architecture is evaluated on Omniglot 5-way 1-shot recognition and CORe50 continual object recognition. With Swin-Tiny, the hierarchical model reaches 97.39\% accuracy on Omniglot and 95.37\% final accuracy on CORe50 when combined with experience replay. Learned multi-bank retrieval reaches 47.50\% delayed-association accuracy, compared with 24.17\% for a single persistent bank and 25.00\% without memory. After intervening distractors, Episodic Memory retains approximately 0.96 cosine similarity with stored associations, while Working Memory falls to approximately 0.05. These results show that Hebbian association and learned memory routing can jointly organize online visual experience across rapid, persistent, and consolidated memory timescales within Vision Transformers.

Problem

• Vision Transformers
• memory timescales
• newly acquired information

Innovation

• Hierarchical Hebbian Memory
• memory timescales
• Vision Transformers
• learned controller
• causal read-before-write lifecycle