MIBench: Evaluating LMMs on Multimodal Interaction

2026-03-12

AI Summary

Current large multimodal language models (LMMs) lack systematic evaluation of their multimodal interaction capabilities. To address this gap, this work proposes MIBench, a structured benchmark that, for the first time, assesses LMMs along two dimensions—modality source bias and multimodal collaborative generation—across three cognitive levels: recognition, comprehension, and reasoning. The benchmark comprises 32 task categories and over 10,000 sample pairs, organized into a unified framework (con_v, con_t, task). Evaluation using MIBench reveals pervasive issues in existing LMMs, including strong text-dominant bias and weak collaborative generation ability. Notably, even native multimodal models exhibit fundamental deficiencies in basic interaction mechanisms. These findings provide critical insights and concrete directions for future research in multimodal interaction modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

In different multimodal scenarios, it needs to integrate and utilize information across modalities in a specific way based on the demands of the task. Different integration ways between modalities are referred to as "multimodal interaction". How well a model handles various multimodal interactions largely characterizes its multimodal ability. In this paper, we introduce MIBench, a comprehensive benchmark designed to evaluate the multimodal interaction capabilities of Large Multimodal Models (LMMs), which formulates each instance as a (con_v , con_t, task) triplet with contexts from vision and text, necessitating that LMMs employ correct forms of multimodal interaction to effectively complete the task. MIBench assesses models from three key aspects: the ability to source information from vision-centric or text-centric cues, and the ability to generate new information from their joint synergy. Each interaction capability is evaluated hierarchically across three cognitive levels: Recognition, Understanding, and Reasoning. MIBench comprises over 10,000 vision-text context pairs spanning 32 distinct tasks. Evaluation of state-of-the-art LMMs show that: (1) LMMs' ability on multimodal interaction remains constrained, despite the scaling of model parameters and training data; (2) they are easily distracted by textual modalities when processing vision information; (3) they mostly possess a basic capacity for multimodal synergy; and (4) natively trained multimodal models show noticeable deficits in fundamental interaction ability. We expect that these observations can serve as a reference for developing LMMs with more enhanced multimodal ability in the future.

Problem

• multimodal interaction
• Large Multimodal Models
• vision-text integration
• multimodal evaluation
• cross-modal synergy

Innovation

• multimodal interaction
• MIBench
• large multimodal models
• cross-modal integration
• hierarchical evaluation