OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

2026-07-09

AI Summary

This work addresses a critical limitation in existing document understanding benchmarks, which often reduce visual content to text and thus fail to assess models’ genuine visual reasoning capabilities. To this end, the authors introduce MapQA, the first map-centric benchmark explicitly designed to evaluate irreducible visual reasoning. It comprises nine map categories, 1,603 images, and 2,096 human-annotated question-answer pairs. The study proposes a Visual Dependency Index (VDI) to quantify the extent to which questions inherently rely on visual information, and validates the benchmark’s high visual dependency through both human annotations and large-scale evaluations with vision-language models (LVLMs). Experiments on 25 state-of-the-art LVLMs reveal a peak accuracy of only 75.03%, substantially lower than performance on other benchmarks, underscoring MapQA’s challenge to current models.

Abstract

Recent advancements in LVLMs necessitate robust benchmarks for complex, visually grounded reasoning. A critical limitation is identified in many document understanding benchmarks: visual content is often reducible to text, enabling high performance without genuine visual grounding. To address this limitation, OmniMapBench is introduced to foster visual-centric reasoning for map documents. The benchmark comprises 2,096 manually annotated question-answer pairs across 1,603 map documents from nine categories. It is designed to probe a hierarchy of skills, ranging from perception to multi-step visual reasoning. To quantify benchmark properties, a simple yet effective benchmark-level metric is proposed: the Visual Dependency Index (VDI), defined as the accuracy drop when images are replaced with question-agnostic descriptions. OmniMapBench exhibits higher VDI than established benchmarks, which quantitatively validates its focus on irreducible visual reasoning. Comprehensive evaluations of 25 leading LVLMs are conducted on OmniMapBench. A significant performance gap is observed, with the top-performing model achieving only 75.03\% accuracy. This result underscores the challenges posed by OmniMapBench to current LVLMs. This work aims to catalyze progress in visual-centric reasoning for document understanding of LVLMs. The dataset and code are publicly available at https://github.com/SIGMME/OmniMapBench.

Problem

• visual-centric reasoning
• map documents
• document understanding
• visual grounding
• benchmark

Innovation

• Visual-Centric Reasoning
• Visual Dependency Index
• Map Document Understanding
• LVLM Benchmarking
• Irreducible Visual Reasoning