MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models

2025-06-05

AI Summary

This work addresses the evaluation bottleneck in assessing multimodal large language models’ (MLLMs) error detection and correction capabilities. To this end, we introduce MMRefine—the first benchmark specifically designed to evaluate MLLMs’ refinement ability—covering six representative multimodal error types. We propose a fine-grained, error-type–aware evaluation framework that moves beyond conventional coarse-grained accuracy-based input-output comparisons. Our methodology incorporates multimodal prompt engineering, controllable error injection, and a structured protocol to ensure reproducible, model-agnostic assessment across both open- and closed-weight MLLMs. Experimental results reveal critical weaknesses in MLLMs’ cross-modal consistency verification and visual-evidence grounding—key components of robust multimodal reasoning—thereby providing empirical evidence and concrete directions for enhancing MLLM inference capabilities. (149 words)

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper introduces MMRefine, a MultiModal Refinement benchmark designed to evaluate the error refinement capabilities of Multimodal Large Language Models (MLLMs). As the emphasis shifts toward enhancing reasoning during inference, MMRefine provides a framework that evaluates MLLMs' abilities to detect and correct errors across six distinct scenarios beyond just comparing final accuracy before and after refinement. Furthermore, the benchmark analyzes the refinement performance by categorizing errors into six error types. Experiments with various open and closed MLLMs reveal bottlenecks and factors impeding refinement performance, highlighting areas for improvement in effective reasoning enhancement. Our code and dataset are publicly available at https://github.com/naver-ai/MMRefine.

Problem

• Evaluates MLLMs' error detection and correction abilities
• Analyzes refinement performance across six error types
• Identifies bottlenecks hindering effective reasoning enhancement

Innovation

• Multimodal benchmark for error refinement evaluation
• Categorizes errors into six distinct types
• Identifies bottlenecks in refinement performance