Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents

2026-01-07

arXiv.org

AI Summary

Existing benchmarks struggle to effectively evaluate the ability of multimodal large language models to retain, organize, and evolve visual and textual information over extended dialogues. To address this gap, this work proposes Mem-Gallery—the first comprehensive benchmark specifically designed for assessing multimodal long-term dialogue memory. It features high-quality, multi-turn image-text conversations and introduces a three-dimensional evaluation framework encompassing memory retrieval and adaptation, memory-based reasoning, and knowledge management. Evaluations of 13 representative memory systems using Mem-Gallery reveal critical bottlenecks in current models’ capacity for organizing and reasoning over multimodal memories, underscoring the necessity of explicitly preserving and structurally organizing multimodal information to support robust long-term dialogue capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Long-term memory is a critical capability for multimodal large language model (MLLM) agents, particularly in conversational settings where information accumulates and evolves over time. However, existing benchmarks either evaluate multi-session memory in text-only conversations or assess multimodal understanding within localized contexts, failing to evaluate how multimodal memory is preserved, organized, and evolved across long-term conversational trajectories. Thus, we introduce Mem-Gallery, a new benchmark for evaluating multimodal long-term conversational memory in MLLM agents. Mem-Gallery features high-quality multi-session conversations grounded in both visual and textual information, with long interaction horizons and rich multimodal dependencies. Building on this dataset, we propose a systematic evaluation framework that assesses key memory capabilities along three functional dimensions: memory extraction and test-time adaptation, memory reasoning, and memory knowledge management. Extensive benchmarking across thirteen memory systems reveals several key findings, highlighting the necessity of explicit multimodal information retention and memory organization, the persistent limitations in memory reasoning and knowledge management, as well as the efficiency bottleneck of current models.

Problem

• multimodal long-term memory
• conversational memory
• MLLM agents
• memory benchmarking
• multimodal understanding

Innovation

• multimodal long-term memory
• MLLM agents
• conversational benchmark
• memory reasoning
• memory knowledge management