Large Language-Geometry Model: When LLM meets Equivariance

2025-02-16

AI Summary

This work addresses the challenge of jointly achieving semantic understanding and geometric equivariance in 3D physical system modeling—specifically for structural and dynamical prediction. We propose EquiLLM, the first language-geometry co-designed architecture. It decouples modeling responsibilities: a large language model (LLM) performs invariant semantic reasoning, while an E(3)-equivariant graph neural network—augmented with an equivariant feature adapter—exclusively handles 3D spatial relationships, guaranteeing strict E(3) equivariance. Geometric-aware prompt engineering enables precise multimodal alignment between linguistic and geometric representations. Evaluated on molecular dynamics simulation, human motion prediction, and antibody design, EquiLLM significantly outperforms state-of-the-art methods, demonstrating strong cross-scientific generalization. To our knowledge, this is the first framework that unifies the semantic reasoning capabilities of LLMs with the geometric rigor of equivariant GNNs in a principled manner.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Accurately predicting 3D structures and dynamics of physical systems is crucial in scientific applications. Existing approaches that rely on geometric Graph Neural Networks (GNNs) effectively enforce $mathrm{E}(3)$-equivariance, but they often fall in leveraging extensive broader information. While direct application of Large Language Models (LLMs) can incorporate external knowledge, they lack the capability for spatial reasoning with guaranteed equivariance. In this paper, we propose EquiLLM, a novel framework for representing 3D physical systems that seamlessly integrates E(3)-equivariance with LLM capabilities. Specifically, EquiLLM comprises four key components: geometry-aware prompting, an equivariant encoder, an LLM, and an equivariant adaptor. Essentially, the LLM guided by the instructive prompt serves as a sophisticated invariant feature processor, while 3D directional information is exclusively handled by the equivariant encoder and adaptor modules. Experimental results demonstrate that EquiLLM delivers significant improvements over previous methods across molecular dynamics simulation, human motion simulation, and antibody design, highlighting its promising generalizability.

Problem

• Predicting 3D structures accurately
• Integrating equivariance with LLM capabilities
• Improving molecular and human motion simulations

Innovation

• Integrates E(3)-equivariance with LLM
• Uses geometry-aware prompting
• Combines equivariant encoder with adaptor