Scaling Autoregressive Models for Lattice Thermodynamics

2026-03-15

AI Summary

Traditional Markov chain Monte Carlo methods suffer from slow convergence and critical slowing down in lattice thermodynamic sampling, while existing autoregressive models incur high memory and training costs due to fixed generation orders, limiting scalability. This work proposes Marginalized Autoregressive Models (MAMs) based on the Transformer architecture, which uniquely integrate arbitrary-order generation with marginal probability modeling and introduce lattice-aware positional encodings to enable flexible generation sequences and cross-scale model transfer. The approach efficiently estimates probabilities of arbitrary partial configurations in a single forward pass, significantly improving free energy accuracy and faithfully capturing phase transition behavior in both the two-dimensional Ising model and CuAu alloy systems. It scales simulations from 10×10 to 20×20 lattices and from 2×2×4 to 4×4×8 supercells at substantially lower computational cost than conventional methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Predicting how materials behave under realistic conditions requires understanding the statistical distribution of atomic configurations on crystal lattices, a problem central to alloy design, catalysis, and the study of phase transitions. Traditional Markov-chain Monte Carlo sampling suffers from slow convergence and critical slowing down near phase transitions, motivating the use of generative models that directly learn the thermodynamic distribution. Existing autoregressive models (ARMs), however, generate configurations in a fixed sequential order and incur high memory and training costs, limiting their applicability to realistic systems. Here, we develop a framework combining any-order ARMs, which generate configurations flexibly by conditioning on any known subset of lattice sites, with marginalization models (MAMs), which approximate the probability of any partial configuration in a single forward pass and substantially reduce memory requirements. This combination enables models trained on smaller lattices to be reused for sampling larger systems, while supporting expressive Transformer architectures with lattice-aware positional encodings at manageable computational cost. We demonstrate that Transformer-based any-order MAMs achieve more accurate free energies than multilayer perceptron-based ARMs on both the two-dimensional Ising model and CuAu alloys, faithfully capturing phase transitions and critical behavior. Overall, our framework scales from $10 \times 10$ to $20 \times 20$ Ising systems and from $2 \times 2 \times 4$ to $4 \times 4 \times 8$ CuAu supercells at reduced computational cost compared to conventional sampling methods.

Problem

• lattice thermodynamics
• autoregressive models
• phase transitions
• generative modeling
• atomic configurations

Innovation

• any-order autoregressive models
• marginalization models
• lattice thermodynamics
• Transformer architectures
• phase transitions