Transformer-Enhanced Variational Autoencoder for Crystal Structure Prediction

2025-02-13

AI Summary

This work addresses crystal structure prediction by proposing the first end-to-end generative model that jointly incorporates equivariant dot-product attention Transformers, adaptive distance expansion, and irreducible group representations to explicitly encode periodicity and spatial symmetries. Built upon a variational autoencoder framework, it enables stable learning of material feature distributions and supports controllable crystal structure reconstruction and generation. The method achieves new state-of-the-art performance on carbon_24, perov_5, and mp_20 benchmarks—reducing structural reconstruction error by 12–28%—while significantly improving physical plausibility (e.g., thermodynamic stability, bond-length/angle validity) and sampling robustness. Its core innovation lies in deeply integrating group representation theory into the equivariant attention mechanism, enabling the first explicit, joint modeling of crystallographic symmetry and translational periodicity within a differentiable deep learning architecture.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Crystal structure forms the foundation for understanding the physical and chemical properties of materials. Generative models have emerged as a new paradigm in crystal structure prediction(CSP), however, accurately capturing key characteristics of crystal structures, such as periodicity and symmetry, remains a significant challenge. In this paper, we propose a Transformer-Enhanced Variational Autoencoder for Crystal Structure Prediction (TransVAE-CSP), who learns the characteristic distribution space of stable materials, enabling both the reconstruction and generation of crystal structures. TransVAE-CSP integrates adaptive distance expansion with irreducible representation to effectively capture the periodicity and symmetry of crystal structures, and the encoder is a transformer network based on an equivariant dot product attention mechanism. Experimental results on the carbon_24, perov_5, and mp_20 datasets demonstrate that TransVAE-CSP outperforms existing methods in structure reconstruction and generation tasks under various modeling metrics, offering a powerful tool for crystal structure design and optimization.

Problem

• Enhance crystal structure prediction accuracy
• Capture periodicity and symmetry effectively
• Improve generative models for material design

Innovation

• Transformer-Enhanced Variational Autoencoder
• Adaptive distance expansion integration
• Equivariant dot product attention mechanism