MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

2026-08-10

AI Summary

Existing benchmarks inadequately assess the capacity of multimodal large language models to reason by integrating dispersed visual evidence with engineering principles in architecture and civil engineering. This work proposes MMArch—the first high-quality multimodal reasoning benchmark tailored to this domain—spanning ten subfields and comprising 1,212 questions derived from figures in peer-reviewed papers. These questions require joint reasoning over cross-figure evidence and domain-specific engineering principles, constructed via a decoupled planning–authoring pipeline. Rigorous quality control combines automated filtering, adversarial auditing, and expert validation. Evaluation across 18 leading models reveals that even the strongest open-source model achieves only ~30% accuracy, while the best closed-source system reaches 52%, both substantially below human experts’ 95%. Error analysis highlights critical deficiencies in applying engineering principles and synthesizing cross-figure evidence.

Abstract

Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion. We introduce MMArch, a benchmark for architecture and civil engineering spanning ten subdomains and built entirely from figures in peer-reviewed papers. Its $1{,}212$ short-answer items are produced by a decoupled planner--writer pipeline and validated through automated screening, a blind adversarial audit, and expert review, so that answering requires perceiving the relevant evidence, identifying the governing principle, and applying it, not exploiting textual or single-figure shortcuts. Evaluating $18$ open-weight and proprietary MLLMs against a domain-expert panel, we find a wide gap: the strongest open-source model attains about $30\%$ and the best proprietary system $52\%$, while human experts reach $95\%$, more than forty points ahead. Our error analysis shows that failures concentrate in applying principles and combining evidence across figures rather than in locating it, pointing to substantial headroom for future research. Code and data are available at https://dcx-swjtu.github.io/MMArch/.

Problem

• multimodal reasoning
• architectural evidence
• engineering principles
• benchmarking
• visual evidence integration

Innovation

• multimodal reasoning
• architectural evidence
• benchmarking
• engineering principles
• MLLM evaluation