Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark

2025-06-04

AI Summary

Existing multimodal large language model (MLLM) benchmarks predominantly focus on single-image reasoning or evaluate only final answers, lacking fine-grained assessment of structured visual reasoning across multiple images. Method: We introduce MMRB—the first benchmark dedicated to multi-image structured visual reasoning—comprising 92 spatial, temporal, and semantic subtasks. We propose a novel stepwise reasoning evaluation paradigm, incorporating multi-solution chain-of-thought (CoT) annotations refined via human curation, and construct a dedicated reward modeling subset for multi-image ordering. Additionally, we release an LLM-based, sentence-level automatic evaluation framework. Results: Evaluations across 40 MLLMs—including 9 reasoning-specialized models—reveal that open-source models significantly underperform commercial counterparts, and current multimodal reward models exhibit near-total failure on multi-image ordering tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

With enhanced capabilities and widespread applications, Multimodal Large Language Models (MLLMs) are increasingly required to process and reason over multiple images simultaneously. However, existing MLLM benchmarks focus either on single-image visual reasoning or on multi-image understanding tasks with only final-answer evaluation, leaving the reasoning capabilities of MLLMs over multi-image inputs largely underexplored. To address this gap, we introduce the $ extbf{Multimodal Multi-image Reasoning Benchmark (MMRB)}$, the first benchmark designed to evaluate structured visual reasoning across multiple images. MMRB comprises $ extbf{92 sub-tasks}$ covering spatial, temporal, and semantic reasoning, with multi-solution, CoT-style annotations generated by GPT-4o and refined by human experts. A derivative subset is designed to evaluate multimodal reward models in multi-image scenarios. To support fast and scalable evaluation, we propose a sentence-level matching framework using open-source LLMs. Extensive baseline experiments on $ extbf{40 MLLMs}$, including 9 reasoning-specific models and 8 reward models, demonstrate that open-source MLLMs still lag significantly behind commercial MLLMs in multi-image reasoning tasks. Furthermore, current multimodal reward models are nearly incapable of handling multi-image reward ranking tasks.

Problem

• Evaluating MLLMs' reasoning over multiple images lacks benchmarks
• Current benchmarks focus on single-image or final-answer tasks
• Multimodal reward models struggle with multi-image ranking tasks

Innovation

• Introduces Multimodal Multi-image Reasoning Benchmark (MMRB)
• Uses GPT-4o and human experts for annotations
• Proposes sentence-level matching framework for evaluation