Model Metamers Reveal Invariances in Graph Neural Networks

2025-10-20

AI Summary

This work identifies **excessive invariance** in Graph Neural Networks (GNNs): node representations exhibit undue robustness to structural and feature perturbations—far exceeding human-perceptible invariance boundaries. To probe this phenomenon, we propose a **node-activation-matching-based graph-level isomer generation method**, synthesizing input graphs that are GNN-representationally equivalent yet structurally or featurally distinct. We further introduce a theoretical framework grounded in **local isomer dimensionality and manifold volume change**, enabling the first quantitative characterization of GNN invariance strength. Experiments reveal that canonical GNNs (e.g., GCN, GAT) suffer from extreme invariance, with existing regularization techniques or architectural modifications offering only partial mitigation. Our work establishes the first invariance-oriented GNN evaluation benchmark, uncovers novel failure modes, and provides a new analytical paradigm for interpretability and robustness research.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In recent years, deep neural networks have been extensively employed in perceptual systems to learn representations endowed with invariances, aiming to emulate the invariance mechanisms observed in the human brain. However, studies in the visual and auditory domains have confirmed that significant gaps remain between the invariance properties of artificial neural networks and those of humans. To investigate the invariance behavior within graph neural networks (GNNs), we introduce a model ``metamers'' generation technique. By optimizing input graphs such that their internal node activations match those of a reference graph, we obtain graphs that are equivalent in the model's representation space, yet differ significantly in both structure and node features. Our theoretical analysis focuses on two aspects: the local metamer dimension for a single node and the activation-induced volume change of the metamer manifold. Utilizing this approach, we uncover extreme levels of representational invariance across several classic GNN architectures. Although targeted modifications to model architecture and training strategies can partially mitigate this excessive invariance, they fail to fundamentally bridge the gap to human-like invariance. Finally, we quantify the deviation between metamer graphs and their original counterparts, revealing unique failure modes of current GNNs and providing a complementary benchmark for model evaluation.

Problem

• Investigating representational invariance gaps in graph neural networks
• Developing metamer generation technique to analyze GNN invariance behavior
• Quantifying deviations between metamer graphs to reveal model limitations

Innovation

• Generating model metamers via graph optimization
• Analyzing metamer manifold volume changes theoretically
• Quantifying invariance gaps between GNNs and humans