Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoning

2026-04-30

AI Summary

Current multimodal large language models (MLLMs) have yet to achieve expert-level performance in perceiving, understanding, and reasoning about scientific experimental images, and a systematic evaluation benchmark is lacking. To address this gap, this work introduces SPUR, a novel benchmark comprising 1,084 expert-curated images and 4,264 question-answer pairs, which for the first time enables systematic assessment across three dimensions: fine-grained panel-level perception, cross-panel relational understanding, and expert-level scientific reasoning. Leveraging six categories of image panels and five experimental paradigms, we evaluate 20 representative MLLMs and four multimodal chain-of-thought (MCoT) approaches, revealing a significant performance gap between existing models and human experts in interpreting scientific imagery. These findings highlight a critical bottleneck in AI for Science.

Abstract

We introduce SPUR, a comprehensive benchmark for scientific experimental image perception, understanding, and reasoning, comprising 4,264 question-answering (QA) pairs derived from 1,084 expert-curated images. SPUR features three key innovations: (1) Panel-Level Fine-Grained Perception: evaluating the visual perception of multimodal large language models (MLLMs) across three dimensions (numerical, morphological, and information localization) on six fine-grained panel types; (2) Cross-Panel Relation Understanding: utilizing complex images with an average of 14.3 panels per sample to evaluate MLLMs' ability to decipher intricate cross-panel relations; (3) Expert-Level Reasoning: assessment of qualitative and quantitative reasoning across five experimental paradigms to determine if models can infer conclusions from evidence as human experts do. Comprehensive evaluation of 20 MLLMs and four multimodal Chain-of-Thought (MCoT) methods reveals that current models fall significantly short of the expert-level requirements for scientific image interpretation, underscoring a critical bottleneck in AI for Science (AI4S) research.

Problem

• scientific experimental images
• multimodal large language models
• visual perception
• cross-panel reasoning
• AI for Science

Innovation

• fine-grained perception
• cross-panel relation understanding
• expert-level reasoning
• scientific image benchmark
• multimodal large language models