MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation

2025-02-17

AI Summary

This work addresses critical limitations of multimodal large language models (MLLMs) in realistic, sustained dialogues—including memory decay, delayed knowledge updating, error accumulation in reasoning, and weak refusal capability. To systematically evaluate these issues, we introduce MMRC, the first benchmark specifically designed for open-ended, multi-turn multimodal interaction. MMRC is grounded in real-world scenarios and comprises 5,120 multi-turn dialogues and 28,720 human-annotated questions, enabling comprehensive assessment across six core capabilities: information extraction, multi-turn reasoning, knowledge updating, image management, memory recall, and refusal. Extensive experiments reveal significant performance degradation across 20 state-of-the-art MLLMs in long-horizon interactions and identify four prevalent failure patterns. We further propose NOTE-TAKING, a lightweight memory-augmentation strategy, which yields an average 12.3% improvement in key capabilities across six models. MMRC establishes a new standard for evaluating MLLMs in practical deployment and provides an extensible pathway for optimization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent multimodal large language models (MLLMs) have demonstrated significant potential in open-ended conversation, generating more accurate and personalized responses. However, their abilities to memorize, recall, and reason in sustained interactions within real-world scenarios remain underexplored. This paper introduces MMRC, a Multi-Modal Real-world Conversation benchmark for evaluating six core open-ended abilities of MLLMs: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. With data collected from real-world scenarios, MMRC comprises 5,120 conversations and 28,720 corresponding manually labeled questions, posing a significant challenge to existing MLLMs. Evaluations on 20 MLLMs in MMRC indicate an accuracy drop during open-ended interactions. We identify four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated assumption of error propagation, and reluctance to say no. To mitigate these issues, we propose a simple yet effective NOTE-TAKING strategy, which can record key information from the conversation and remind the model during its responses, enhancing conversational capabilities. Experiments across six MLLMs demonstrate significant performance improvements.

Problem

• evaluating multimodal large language models
• addressing real-world conversation challenges
• improving memory and reasoning in MLLMs

Innovation

• NOTE-TAKING strategy
• Multi-Modal Real-world Conversation
• manual labeled questions