MMGist: A Comprehensive Multimodal Benchmark for 2027

2026-06-21

AI Summary

Existing vision-language benchmarks often suffer from weak visual dependency, performance saturation, and anomalous samples, limiting their ability to effectively evaluate the true capabilities of multimodal models. To address these issues, this work proposes MMGist—a high-quality multimodal benchmark constructed through a three-stage filtering pipeline involving text ablation, cross-model saturation analysis, and anomaly detection. From an initial pool of 23,250 samples, the pipeline yields 7,262 highly discriminative, visually grounded, and reliable evaluation items spanning seven core competency dimensions. Experiments across 27 state-of-the-art large vision-language models demonstrate that MMGist achieves a 69% reduction in item count while preserving model ranking fidelity at 0.98 and enhancing cross-model discriminability by 78%, thereby revealing systematic deficiencies in current models’ visual reasoning abilities.

Abstract

We conduct a systematic study of 18 widely used vision-language benchmarks and identify three major issues: 1) many items do not rely on visual cues and therefore fail to effectively measure multimodal understanding; 2) many items are already close to performance saturation for current LVLMs, which limits their discriminative power; 3) a small number of anomalous items affect the reliability of evaluation results. To this end, we propose MMGist, a curated benchmark that covers seven capability dimensions and contains 7,262 items. MMGist is constructed through a three-stage pipeline, which sequentially combines text-ablation filtering, cross-model saturation filtering, and anomaly detection filtering. We conduct extensive experiments on 27 leading LVLMs and compare MMGist with the raw pool of 23,250 items. The results show that MMGist preserves model rankings with high fidelity, with Spearman $ρ= 0.98$, while reducing evaluation items by 69\% and improving cross-model discrimination by 78\%. Further results indicate that Visual Logic remains a systematic weakness of current LVLMs, while knowledge-intensive dimensions such as Expert Knowledge dimensions remain important factors for distinguishing closed-source models from open-source models. These findings suggest that high-quality evaluation should prioritize visual dependency, discriminative power, and reliability, rather than simply pursuing benchmark scale.

Problem

• multimodal benchmark
• visual-language models
• evaluation reliability
• performance saturation
• visual dependency

Innovation

• multimodal benchmark
• vision-language models
• evaluation reliability
• discriminative power
• visual dependency