VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models

2025-11-14

AI Summary

Vision-language models (VLMs) suffer from a “visual processing bottleneck” in complex vision tasks, leading to loss of visual evidence and insufficient contextualized perception. To address this, we propose VisMem—a novel framework that, for the first time, integrates human cognitive memory theory into VLMs. VisMem introduces a dual-module architecture: a short-term perceptual memory module preserving fine-grained visual features, and a long-term semantic memory module modeling abstract conceptual associations. Both modules operate in the latent space, enabling dynamic storage, selective activation, and adaptive fusion of visual memories. This synergistic design enhances perceptual fidelity and semantic consistency during multimodal reasoning and generation. Evaluated across diverse vision understanding, reasoning, and generation benchmarks, VisMem achieves an average performance gain of 11.8% over strong baselines, significantly outperforming existing methods. Our work establishes a new paradigm for latent-space memory augmentation in VLMs.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a"visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.8% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git.

Problem

• VLMs lose visual grounding during prolonged generation tasks
• VLMs lack contextualized visual experience across complex tasks
• Visual processing bottleneck limits VLM performance on complex tasks

Innovation

• Dynamic latent vision memories enhance VLMs
• Short-term module retains fine-grained perceptual details
• Long-term module consolidates abstract semantic information