ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models

2026-06-26

AI Summary

This work addresses the limited adaptability of current vision-language models in dynamic test-time environments, stemming from their lack of cross-modal knowledge accumulation and long-term memory mechanisms. Inspired by the brain’s complementary memory systems, the authors propose a novel dual-memory architecture that integrates a hippocampus-like module—rapidly caching high-confidence visual samples—and a neocortex-like module—slowly consolidating global textual prototypes. These components are jointly optimized through cross-modal alignment to ensure consistency during test-time adaptation. Evaluated across 15 benchmark datasets, the method significantly outperforms existing approaches, achieving state-of-the-art performance under both natural distribution shifts and cross-dataset generalization scenarios, thereby enabling effective multimodal synergy and continual knowledge integration.

Abstract

Test-time adaptation (TTA) of vision-language models (VLMs) is essential for their robust deployment in dynamic, real-world environments. However, existing TTA methods often adapt locally without accumulating knowledge over time, or operating within a single modality without exploiting VLMs' inherently multi-modal nature. Inspired by the \textbf{Com}plementary \textbf{Mem}ory systems of the biological brain, we propose \textbf{ComMem}, an innovative approach that mimics the distinct but cooperative roles of the hippocampus and neocortex to enable effective TTA for VLMs. ComMem consists of two key components: a fast-adapting detailed memory, akin to the hippocampus, that forms a dynamic visual cache from high-confidence test samples; and a slow-integrating abstract memory, akin to the neocortex, that continually refines global textual prototypes. For each test instance, ComMem jointly optimizes both memory systems to ensure cross-modal consistency. Extensive experiments on 15 benchmark datasets show that ComMem significantly outperforms state-of-the-art methods under both natural distribution shifts and cross-dataset generalization, offering a promising direction for enhancing VLMs' practical adaptability.

Problem

• Test-time adaptation
• Vision-language models
• Multi-modal learning
• Distribution shift
• Model robustness

Innovation

• Test-Time Adaptation
• Vision-Language Models
• Complementary Memory Systems
• Cross-Modal Consistency
• Dynamic Visual Cache