From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals

2026-05-28

AI Summary

This work addresses a key bottleneck in materials discovery—the lack of reliable and interpretable evaluation methods for the vast space of candidate materials—by introducing MaterEval, a novel framework that transforms expert rule systems into learnable preference signals. Specifically, for each material, the framework generates paired judgments: one grounded in domain-specific rules and another produced without such guidance, thereby constructing preference pairs to fine-tune small, open-source large language models. Integrating a dual-process reasoning mechanism that combines fast and slow inference pathways, MaterEval significantly enhances model performance on high-entropy alloy assessment tasks, achieving marked improvements in accuracy, conclusion consistency, and evidence discriminability—all without external retrieval. The resulting system approaches the performance of closed-source, rule-driven models while offering efficient, knowledge-enhanced, and interpretable material evaluations.

Abstract

As candidate generation and high-throughput experimentation advance, the primary bottleneck in materials discovery is shifting from property prediction to making reliable evaluations among massive candidate sets. We propose a Knowledge-Augmented Preference Signals Framework, MaterEval, that automatically produces, for the same candidate, two evaluations: an informed judgment that follows expert rules and provides supporting evidence, and a rule-removed blind guess. By pairing the two evaluations as preference data, we guide general-purpose large language models (LLMs), originally lacking materials-specific criteria, from intuitive judgment toward reliable evaluation supported by explicit evidence. To balance throughput, cost, and reliability, we further introduce a fast-slow reasoning scheme that decouples large-scale rapid screening from in-depth review on a small subset. Using high-entropy alloy (HEA) assessment as a case study, we show that, without external retrieval and relying solely on internalized capabilities, small open-source LLMs achieve substantial gains in accuracy, conclusion consistency, and evidence discrimination, approaching the performance of rule-based closed-source LLMs. These results demonstrate that expert rules can be systematically transformed into learnable preference signals, enabling a low-cost and deployable evaluation module for autonomous materials discovery loops.

Problem

• materials discovery
• candidate evaluation
• reliable assessment
• high-throughput experimentation
• preference signals

Innovation

• Knowledge-Augmented Preference Signals
• Large Language Models
• Materials Evaluation
• Fast-Slow Reasoning
• Autonomous Discovery