Adaptive Hebbian Memory Routing in Vision Transformers for Few-Shot Learning

2026-06-23

AI Summary

This work addresses the challenge of few-shot image recognition, where models struggle to rapidly adapt to novel classes given limited samples. To this end, the authors introduce a lightweight MLP-based routing mechanism into Vision Transformers to dynamically modulate the activation, update, and retention strength of Hebbian memory. They propose three adaptive strategies—adaptive placement, adaptive plasticity, and fully adaptive Hebbian routing—that overcome the limitations of conventional fixed-memory behaviors. Experiments on ViT-Small, DeiT-Small, and Swin-Tiny architectures demonstrate consistent improvements over fixed Hebbian approaches on Omniglot and CIFAR-FS benchmarks. Notably, the Swin-based model achieves 96.94% accuracy while reducing inference time from 16.51 ms to 14.05 ms, significantly enhancing both generalization capability and computational efficiency.

Abstract

Few-shot image recognition requires models to adapt to new classes from a small labeled support set. Hebbian fast-weight memory can provide temporary associative information during an episode, but fixed memory behavior may not be appropriate for every few-shot task. In this work, we propose Adaptive Hebbian Routing for few-shot Vision Transformers. The method uses a lightweight MLP router to control the contribution of Hebbian memory, the strength of memory updates, and the retention of previous memory from support-set features. We study Adaptive Placement, Adaptive Plasticity, and Fully Adaptive Hebbian Routing. Experiments use ViT-Small, DeiT-Small, and Swin-Tiny under 5-way 1-shot evaluation on Omniglot, CIFAR-FS, and cross-domain transfer from CIFAR-FS to Omniglot. In the direct Swin comparison, fixed and adaptive Hebbian variants use the same memory location. Adaptive Plasticity improves the fixed Hebbian result from 96.74\% to 96.92\%, while Fully Adaptive Routing achieves the best result at 96.94\%. The fully adaptive Swin model also reduces inference time from 16.51 ms to 14.05 ms relative to fixed Hebbian Swin. On CIFAR-FS, adaptive variants improve performance across all three backbones, and the multi-shot evaluation shows that these gains remain useful as the number of support examples increases. These results show that adaptive plasticity and adaptive memory activation can improve few-shot Transformer representations beyond fixed Hebbian behavior.

Problem

• Few-shot learning
• Vision Transformers
• Hebbian memory
• Adaptive routing
• Image recognition

Innovation

• Adaptive Hebbian Routing
• Few-Shot Learning
• Vision Transformers
• Hebbian Memory
• Memory Plasticity