Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?

2024-10-15

Neural Information Processing Systems

AI Summary

This work challenges the prevailing assumption that E(3)-equivariant graph neural networks (E(3)-GNNs) require only first-order steerable vectors. We theoretically demonstrate that, under k-fold rotational symmetry and polyhedral symmetries, relying solely on first-order steerable vectors collapses model expressivity to the zero function. We provide the first rigorous proof of the necessity of higher-order steerable vectors for faithful E(3)-equivariance. To address this, we propose HEGNN—a novel, efficient E(3)-equivariant architecture integrating higher-order spherical harmonics with scalarized message passing, enabling equivariant updates with steerable vectors of arbitrary order. We empirically verify the theoretical expressivity collapse on symmetric synthetic datasets. On N-body and MD17 benchmarks, HEGNN consistently outperforms EGNN in both energy and force prediction accuracy, achieving superior expressivity without sacrificing computational efficiency.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Equivariant Graph Neural Networks (GNNs) that incorporate E(3) symmetry have achieved significant success in various scientific applications. As one of the most successful models, EGNN leverages a simple scalarization technique to perform equivariant message passing over only Cartesian vectors (i.e., 1st-degree steerable vectors), enjoying greater efficiency and efficacy compared to equivariant GNNs using higher-degree steerable vectors. This success suggests that higher-degree representations might be unnecessary. In this paper, we disprove this hypothesis by exploring the expressivity of equivariant GNNs on symmetric structures, including $k$-fold rotations and regular polyhedra. We theoretically demonstrate that equivariant GNNs will always degenerate to a zero function if the degree of the output representations is fixed to 1 or other specific values. Based on this theoretical insight, we propose HEGNN, a high-degree version of EGNN to increase the expressivity by incorporating high-degree steerable vectors while maintaining EGNN's efficiency through the scalarization trick. Our extensive experiments demonstrate that HEGNN not only aligns with our theoretical analyses on toy datasets consisting of symmetric structures, but also shows substantial improvements on more complicated datasets such as $N$-body and MD17. Our theoretical findings and empirical results potentially open up new possibilities for the research of equivariant GNNs.

Problem

• Investigating if high-degree representations are necessary in equivariant graph neural networks
• Demonstrating theoretical limitations of low-degree representations on symmetric structures
• Proposing high-degree EGNN to enhance expressivity while maintaining efficiency

Innovation

• Incorporates high-degree steerable vectors for enhanced expressivity
• Maintains efficiency using scalarization trick from EGNN
• Proposes HEGNN model to handle symmetric structures effectively