More Images, More Problems? A Controlled Analysis of VLM Failure Modes

2026-01-12

AI Summary

Current large vision-language models struggle to effectively aggregate cross-image information and track multiple concepts in multi-image input scenarios, limiting their reasoning capabilities. To address this, this work introduces MIMIC—the first controllable evaluation benchmark specifically designed for multi-image understanding—which systematically reveals failure modes of existing models. Furthermore, the authors propose a procedural multi-image data synthesis approach, an attention mask mechanism tailored for multi-image inputs, and a multi-image instruction tuning strategy. Experimental results demonstrate that the proposed method significantly enhances the model’s ability to integrate cross-image information, outperforming state-of-the-art approaches across multiple benchmarks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored. While existing benchmarks have initiated the evaluation of multi-image models, a comprehensive analysis of their core weaknesses and their causes is still lacking. In this work, we introduce MIMIC (Multi-Image Model Insights and Challenges), a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs. Using MIMIC, we conduct a series of diagnostic experiments that reveal pervasive issues: LVLMs often fail to aggregate information across images and struggle to track or attend to multiple concepts simultaneously. To address these failures, we propose two novel complementary remedies. On the data side, we present a procedural data-generation strategy that composes single-image annotations into rich, targeted multi-image training examples. On the optimization side, we analyze layer-wise attention patterns and derive an attention-masking scheme tailored for multi-image inputs. Experiments substantially improved cross-image aggregation, while also enhancing performance on existing multi-image benchmarks, outperforming prior state of the art across tasks. Data and code will be made available at https://github.com/anurag-198/MIMIC.

Problem

• Large Vision Language Models
• multi-image understanding
• failure modes
• cross-image reasoning
• visual language models

Innovation

• Multi-image reasoning
• Vision-language models
• Attention masking
• Procedural data generation
• Failure mode analysis