Solving Over-Smoothing in GNNs via Nonlocal Message Passing: Algebraic Smoothing and Depth Scalability

2025-12-09

AI Summary

The relationship between layer normalization (LN) placement and oversmoothing in graph neural networks (GNNs) remains unclear: Pre-LN mitigates oversmoothing but suffers from the depth curse, whereas Post-LN alleviates the depth curse yet exacerbates oversmoothing. Method: We identify a fundamental trade-off between smoothing control and depth scalability, and propose a Post-LN-based non-local message passing mechanism. Leveraging algebraic smoothing theory, our approach enables controllable information propagation without introducing additional parameters. Contribution/Results: Guided by theoretically grounded normalization dynamics analysis, our method achieves both deep scalability and oversmoothing suppression. Empirical evaluation on five benchmark datasets demonstrates effectiveness—supporting GNNs up to 256 layers—while significantly improving performance and maintaining model efficiency.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

The relationship between Layer Normalization (LN) placement and the over-smoothing phenomenon remains underexplored. We identify a critical dilemma: Pre-LN architectures avoid over-smoothing but suffer from the curse of depth, while Post-LN architectures bypass the curse of depth but experience over-smoothing. To resolve this, we propose a new method based on Post-LN that induces algebraic smoothing, preventing over-smoothing without the curse of depth. Empirical results across five benchmarks demonstrate that our approach supports deeper networks (up to 256 layers) and improves performance, requiring no additional parameters. Key contributions: Theoretical Characterization: Analysis of LN dynamics and their impact on over-smoothing and the curse of depth. A Principled Solution: A parameter-efficient method that induces algebraic smoothing and avoids over-smoothing and the curse of depth. Empirical Validation: Extensive experiments showing the effectiveness of the method in deeper GNNs.

Problem

• Addresses over-smoothing in GNNs via nonlocal message passing
• Resolves dilemma between over-smoothing and curse of depth
• Enables deeper GNNs without extra parameters via algebraic smoothing

Innovation

• Post-LN based method induces algebraic smoothing
• Prevents over-smoothing without depth curse
• Supports deep networks up to 256 layers