MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understanding

2026-07-03

AI Summary

This study addresses the lack of systematic evaluation of current vision-language models on complex materials phase diagram understanding, which demands thermodynamic reasoning. The authors construct the first high-fidelity multimodal benchmark specifically tailored for phase diagrams, comprising 200 carefully curated image–text pairs extracted from 3,681 scientific publications. Designed to support open-ended, deep scientific image understanding tasks, this benchmark overcomes the limitations of conventional objective-question-based evaluations by integrating literature mining, semantic alignment, and expert-level human validation. Comprehensive assessments reveal that existing models exhibit only superficial perceptual capabilities, significantly underperforming human experts in fine-grained recognition and multi-diagram scenarios, and remain incapable of performing deep mechanistic reasoning required for accurate phase diagram interpretation.

Abstract

Materials phase diagrams are a core knowledge representation in materials science, encoding temperature,composition, phase stability, and phase transformation pathways, with their full understanding requiring thermodynamic mechanism analysis and scientific reasoning. Although VLMs have shown promise in scientific image understanding, their systematic evaluation on such logically complex images demanding deep mechanistic interpretation remains limited, and phase diagrams provide a challenging testbed for this purpose. We introduce MatPhaseBench, a high-quality, high-reliability benchmark for complex scientific image understanding, focused on materials phase diagrams. MatPhaseBench is constructed from 3681 papers in classical materials science journals, from which 200 high-quality diagram-text pairs were selected, covering 189 material systems and 70 elements. The benchmark has three key features: (1)targeting complex scientific image understanding-it moves beyond simple objective tests to open-ended tasks requiring deep comprehension; (2)comprehensive image-text alignment-semantic information associated with images is fully preserved during literature mining and matching; (3) high-quality human-supervised text acquisition-all descriptions undergo strict manual validation. Experimental results show that current VLMs remain substantially behind expert-level understanding: they are largely limited to surface visual perception, lack deep reasoning grounded in thermodynamic mechanisms, have limited domain awareness and expert analytical experience, and perform poorly in distinguishing fine-grained differences in composite or multi-diagram settings. Overall, MatPhaseBench constitutes a challenging research-grade benchmark, providing a foundational platform for complex scientific image understanding, phase diagram analysis, and trustworthy multi-modal AI in science.

Problem

• materials phase diagrams
• visual language models
• scientific image understanding
• thermodynamic reasoning
• benchmark evaluation

Innovation

• phase diagram understanding
• scientific image benchmark
• visual language models
• semantic alignment
• thermodynamic reasoning