Foundational Large Language Models for Materials Research

2024-12-12

arXiv.org

AI Summary

To address the challenges of information overload and inefficient knowledge extraction from materials science literature—which hinder research progress—this study introduces LLaMat, a domain-specific foundational large language model for materials science. LLaMat is developed via continued pretraining of the LLaMA architecture on large-scale materials literature and crystallographic CIF data, enabling deep adaptation for text understanding, structured information extraction, and crystal structure generation. We first identify the critical impact of base model selection, demonstrating that LLaMA-2 outperforms LLaMA-3 in domain-specific performance. We further propose LLaMat-CIF, a specialized variant achieving both high stability and broad coverage in periodic-table-scale crystal generation, overcoming rigidity limitations of conventional domain adaptation. Experiments show that LLaMat significantly surpasses general-purpose LLMs on materials NLP and information extraction tasks; LLaMat-CIF successfully generates thousands of thermodynamically stable novel crystals, covering over 90% of chemical elements, with a 42% improvement in structural prediction accuracy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Materials discovery and development are critical for addressing global challenges. Yet, the exponential growth in materials science literature comprising vast amounts of textual data has created significant bottlenecks in knowledge extraction, synthesis, and scientific reasoning. Large Language Models (LLMs) offer unprecedented opportunities to accelerate materials research through automated analysis and prediction. Still, their effective deployment requires domain-specific adaptation for understanding and solving domain-relevant tasks. Here, we present LLaMat, a family of foundational models for materials science developed through continued pretraining of LLaMA models on an extensive corpus of materials literature and crystallographic data. Through systematic evaluation, we demonstrate that LLaMat excels in materials-specific NLP and structured information extraction while maintaining general linguistic capabilities. The specialized LLaMat-CIF variant demonstrates unprecedented capabilities in crystal structure generation, predicting stable crystals with high coverage across the periodic table. Intriguingly, despite LLaMA-3's superior performance in comparison to LLaMA-2, we observe that LLaMat-2 demonstrates unexpectedly enhanced domain-specific performance across diverse materials science tasks, including structured information extraction from text and tables, more particularly in crystal structure generation, a potential adaptation rigidity in overtrained LLMs. Altogether, the present work demonstrates the effectiveness of domain adaptation towards developing practically deployable LLM copilots for materials research. Beyond materials science, our findings reveal important considerations for domain adaptation of LLMs, such as model selection, training methodology, and domain-specific performance, which may influence the development of specialized scientific AI systems.

Problem

• Material Science
• Literature Analysis
• Knowledge Discovery

Innovation

• LLaMat
• Materials Science
• Crystal Structure Generation