Towards Fully-Automated Materials Discovery via Large-Scale Synthesis Dataset and Expert-Level LLM-as-a-Judge

2025-02-23

AI Summary

Material synthesis has long relied on empirical trial-and-error, hindering innovation in energy, catalysis, and biomedicine. To address this, we introduce AlchemyBench—the first LLM-specific benchmark for end-to-end materials synthesis—comprising 17K expert-validated synthesis protocols. It supports three core tasks: precursor/equipment prediction, synthetic procedure generation, and characterization outcome forecasting. We propose LLM-as-a-Judge, an automated evaluation framework achieving a Pearson correlation of 0.89 with human expert scores and statistical agreement comparable to domain experts. Our approach pioneers end-to-end LLM-based synthesis intelligence via large-scale synthesis data, multi-stage task modeling, and prompt engineering. AlchemyBench fills a critical gap by providing the first high-quality, expert-curated benchmark for materials synthesis, significantly enhancing both the efficiency and reliability of synthesis protocol generation and assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Materials synthesis is vital for innovations such as energy storage, catalysis, electronics, and biomedical devices. Yet, the process relies heavily on empirical, trial-and-error methods guided by expert intuition. Our work aims to support the materials science community by providing a practical, data-driven resource. We have curated a comprehensive dataset of 17K expert-verified synthesis recipes from open-access literature, which forms the basis of our newly developed benchmark, AlchemyBench. AlchemyBench offers an end-to-end framework that supports research in large language models applied to synthesis prediction. It encompasses key tasks, including raw materials and equipment prediction, synthesis procedure generation, and characterization outcome forecasting. We propose an LLM-as-a-Judge framework that leverages large language models for automated evaluation, demonstrating strong statistical agreement with expert assessments. Overall, our contributions offer a supportive foundation for exploring the capabilities of LLMs in predicting and guiding materials synthesis, ultimately paving the way for more efficient experimental design and accelerated innovation in materials science.

Problem

• Automating materials discovery process
• Data-driven synthesis prediction framework
• LLM-based expert-level evaluation system

Innovation

• Large-scale synthesis dataset curation
• LLM-as-a-Judge framework
• End-to-end synthesis prediction framework