AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking

2026-07-13

AI Summary

This work addresses the critical limitation of existing materials property prediction benchmarks—such as MatBench—inadequately evaluating out-of-distribution (OOD) generalization, thereby hindering reliable discovery of novel materials. To overcome this, the authors integrate MatBench with principled OOD evaluation protocols and Bayesian optimization to establish the first configurable, automated framework for benchmark optimization in materials informatics. Their approach systematically uncovers, for the first time, the significant causal influence of benchmark configuration on model performance. Remarkably, the framework reproduces results from conventional benchmarks in just 12 optimization steps, reducing computational costs by over 50%, while simultaneously revealing more fundamental evaluation principles that substantially enhance the efficiency and reliability of new materials discovery.

Abstract

Material property prediction (MPP) infers key properties from chemical composition and structure, accelerating the discovery and optimization of novel materials. In the realm of MPP, MatBench is a widely accepted benchmarking tool that defines over ten significant problems and provides the paradigm of performance evaluation for AI prediction models. Even though MatBench works well in benchmarking the performances of prediction models on in-distribution (ID) tasks and datasets, it lacks the ability to reflect their performances on out-of-distribution (OOD) material data, resulting failure in new material discovery. By combining the pipelines of MatBench and the existing researches on OOD performance evaluation, this study enables a huge space of benchmarking configurations, comprehensively reflecting the performances, abilities, and disadvantages of various AI prediction models. This work reports that the discrepancy of performances at different configuration values is huge and can be illustrated with prior knowledge and novel insights, therefore consideration of causal effect of configurations on performance results is necessary. In case of the impossibility of enumerative benchmarking at every configuration, this work further proposes AutoMatBench, an automatic toolkit with Bayesian optimization. Experiments with AutoMatBench reports that, within twelve steps of optimization, the similar results with MatBench and former OOD research can be accessed while more than half of the cost are saved. Besides, this tool also yields more essential findings on MPP benchmarking, positively contributing to the cost and efficiency of new material discovery.

Problem

• material property prediction
• out-of-distribution
• benchmarking
• MatBench
• AI models

Innovation

• AutoMatBench
• out-of-distribution (OOD)
• Bayesian optimization
• material property prediction
• benchmarking