Scaling Laws and Symmetry, Evidence from Neural Force Fields

2025-10-10

AI Summary

This study investigates the role of symmetry—specifically equivariance—in learning interatomic potentials and its impact on scaling laws for large-scale models. Methodologically, we systematically compare equivariant versus non-equivariant neural network force fields, analyzing power-law relationships among dataset size, model parameter count, and computational cost. Our results show that: (1) explicit incorporation of rotational and translational equivariance significantly improves scaling efficiency, yielding faster decay of test error with increasing compute; (2) higher-order tensor representations further enhance scaling exponents, particularly in the large-model regime; and (3) equivariant architectures reduce intrinsic task difficulty, empirically refuting the hypothesis that models can spontaneously learn symmetries from data. Collectively, these findings demonstrate that computationally optimal training requires coordinated scaling of data and model size, and that architectural symmetry priors—not just scale—are decisive determinants of model scaling behavior.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

We present an empirical study in the geometric task of learning interatomic potentials, which shows equivariance matters even more at larger scales; we show a clear power-law scaling behaviour with respect to data, parameters and compute with ``architecture-dependent exponents''. In particular, we observe that equivariant architectures, which leverage task symmetry, scale better than non-equivariant models. Moreover, among equivariant architectures, higher-order representations translate to better scaling exponents. Our analysis also suggests that for compute-optimal training, the data and model sizes should scale in tandem regardless of the architecture. At a high level, these results suggest that, contrary to common belief, we should not leave it to the model to discover fundamental inductive biases such as symmetry, especially as we scale, because they change the inherent difficulty of the task and its scaling laws.

Problem

• Empirical study shows equivariance matters more at larger scales
• Equivariant architectures scale better than non-equivariant models
• Higher-order representations yield improved scaling exponents

Innovation

• Equivariant architectures scale better than non-equivariant models
• Higher-order representations improve scaling exponents
• Compute-optimal training requires scaling data and models together