Analyzing the Effect of Embedding Norms and Singular Values to Oversmoothing in Graph Neural Networks

2025-10-07

AI Summary

Deep graph neural networks (GNNs) suffer from over-smoothing, where node representations become indistinguishable as network depth increases. Method: This paper introduces the first theoretical framework analyzing over-smoothing via node embedding norms and singular values of weight matrices, revealing its root cause as the coupling between adjacency hops and trainable layer count. We propose MASED (Mean Angular Similarity of Embedding Differences), a novel metric quantifying inter-layer embedding degradation, and design G-Reg—a regularization method that decouples graph propagation from parameter learning. G-Reg integrates MASED-based inter-layer boundary control, embedding norm regularization, and adjacency matrix distribution optimization. Contribution/Results: Experiments demonstrate significant improvements in node classification accuracy for deep GNNs; notably, G-Reg outperforms shallow models even in featureless cold-start settings. The work provides both theoretical foundations and practical solutions for scaling GNN depth.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In this paper, we study the factors that contribute to the effect of oversmoothing in deep Graph Neural Networks (GNNs). Specifically, our analysis is based on a new metric (Mean Average Squared Distance - $MASED$) to quantify the extent of oversmoothing. We derive layer-wise bounds on $MASED$, which aggregate to yield global upper and lower distance bounds. Based on this quantification of oversmoothing, we further analyze the importance of two different properties of the model; namely the norms of the generated node embeddings, along with the largest and smallest singular values of the weight matrices. Building on the insights drawn from the theoretical analysis, we show that oversmoothing increases as the number of trainable weight matrices and the number of adjacency matrices increases. We also use the derived layer-wise bounds on $MASED$ to form a proposal for decoupling the number of hops (i.e., adjacency depth) from the number of weight matrices. In particular, we introduce G-Reg, a regularization scheme that increases the bounds, and demonstrate through extensive experiments that by doing so node classification accuracy increases, achieving robustness at large depths. We further show that by reducing oversmoothing in deep networks, we can achieve better results in some tasks than using shallow ones. Specifically, we experiment with a ``cold start" scenario, i.e., when there is no feature information for the unlabeled nodes. Finally, we show empirically the trade-off between receptive field size (i.e., number of weight matrices) and performance, using the $MASED$ bounds. This is achieved by distributing adjacency hops across a small number of trainable layers, avoiding the extremes of under- or over-parameterization of the GNN.

Problem

• Analyzing embedding norms and singular values causing GNN oversmoothing
• Quantifying oversmoothing effects through theoretical distance bounds
• Developing regularization to improve deep GNN robustness and accuracy

Innovation

• Proposed MASED metric to quantify oversmoothing in GNNs
• Introduced G-Reg regularization to increase layer-wise bounds
• Decoupled adjacency hops from weight matrices to reduce oversmoothing