Neighborhood Sampling Does Not Learn the Same Graph Neural Network

2025-09-26

AI Summary

This work addresses the unresolved question of how neighborhood sampling methods systematically affect large-scale graph neural network (GNN) training. We introduce the first unified analytical framework grounded in Neural Tangent Kernel (NTK) and Gaussian Process (GP) theory to model and characterize the impact of diverse sampling strategies on training dynamics and posterior distributions. Our analysis reveals that, while all sampling schemes converge to the same GP posterior as the sample size tends to infinity, they induce markedly distinct posterior covariance structures under finite-sample regimes—leading to incomparable lower bounds on prediction error. Consequently, no universally optimal sampling strategy exists. This work establishes the first theoretical link between sampling behavior and generalization performance in GNNs, providing a principled, NTK/GP-based foundation for sampling design and analysis.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Reasoning under Uncertainty: Graphical Models
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

Neighborhood sampling is an important ingredient in the training of large-scale graph neural networks. It suppresses the exponential growth of the neighborhood size across network layers and maintains feasible memory consumption and time costs. While it becomes a standard implementation in practice, its systemic behaviors are less understood. We conduct a theoretical analysis by using the tool of neural tangent kernels, which characterize the (analogous) training dynamics of neural networks based on their infinitely wide counterparts -- Gaussian processes (GPs). We study several established neighborhood sampling approaches and the corresponding posterior GP. With limited samples, the posteriors are all different, although they converge to the same one as the sample size increases. Moreover, the posterior covariance, which lower-bounds the mean squared prediction error, is uncomparable, aligning with observations that no sampling approach dominates.

Problem

• Analyzes systemic behaviors of neighborhood sampling in GNN training
• Compares posterior Gaussian processes under different sampling approaches
• Reveals no sampling method dominates due to incomparable covariances

Innovation

• Neighborhood sampling reduces exponential growth of neighborhoods
• Theoretical analysis uses neural tangent kernels and Gaussian processes
• Different sampling methods yield incomparable posterior covariances