OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields

2026-05-28

AI Summary

This work addresses the lack of comprehensive benchmarks for evaluating multimodal reasoning capabilities in materials science, as existing assessments predominantly focus on isolated tasks. The study introduces the first multimodal reasoning benchmark spanning 19 subfields, comprising 3,171 expert-calibrated questions that encompass fundamental theory, structural and engineering materials, processing and manufacturing, and functional applications. By integrating expert-curated knowledge graphs, multimodal question design, and human calibration, the benchmark evaluates 13 leading multimodal large language models under settings involving formula understanding, retrieval augmentation, and code assistance. Results reveal that even the best-performing model achieves only a modest overall score of 0.372, highlighting significant deficiencies in cross-subfield reasoning, knowledge integration, and higher-order application—thereby underscoring the benchmark’s critical role in advancing materials intelligence.

Abstract

As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdisciplinary, multimodal, and application-driven nature. However, existing materials benchmarks mainly focus on property prediction, knowledge QA, or characterization understanding, leaving the broader reasoning process from materials knowledge to application underexplored. To fill this gap, we present OmniMatBench, a human-calibrated multimodal reasoning benchmark for materials science. OmniMatBench contains 3,171 expert-curated QA and calculation problems across 19 materials-science subfields, spanning fundamental materials knowledge, structural and engineering materials, materials processing and manufacturing, and functional and applied materials. We evaluate 13 open-source and closed-source MLLMs and find that the best model achieves only a 0.372 overall score, revealing a substantial gap in current materials-science reasoning. Further analysis shows strong variation across subfields, fixed reasoning heuristics, uneven materials knowledge, and limited high-level knowledge application under formula-, retrieval-, and code-assisted settings. OmniMatBench provides crucial insights into the capabilities and limitations of current MLLMs and establishes a foundation for reliable AI assistants in materials-science research.

Problem

• multimodal reasoning
• materials science
• benchmark
• scientific reasoning
• MLLMs

Innovation

• multimodal reasoning
• materials science benchmark
• human-calibrated evaluation
• MLLMs
• cross-subfield assessment