Hyperparameter Transfer in Graph Neural Networks

2026-07-06

AI Summary

This work proposes a hyperparameter transfer framework for graph neural networks (GNNs) tailored to SGD, Adam, and AdamW optimizers, addressing the challenge of cross-scale hyperparameter generalization. For SGD, it introduces a graph-dependent first-layer correction factor; for Adam, it elucidates the critical role of message-passing normalization; and for AdamW, it devises a joint learning rate and weight decay transfer strategy. Theoretical scaling analysis and empirical experiments across varying model widths and depths demonstrate that the proposed approach stabilizes feature updates, enables effective learning rate transfer, and significantly enhances performance as model scale increases.

Abstract

The performance of deep learning models crucially depends on the settings of hyperparameters like learning rate, initialization scale, and weight decay. Hyperparameter transfer aims to make near-optimal hyperparameter settings consistent across model scale, so that large models can be optimized by proxy tuning their smaller, cheaper-to-optimize counterparts. While transfer principles are well-studied in the context of dense neural networks in language and vision tasks, they remain comparatively under-explored for graph neural networks (GNNs). We develop and validate a transfer parameterization for GNNs trained with SGD, Adam, and AdamW. Through theoretical scaling analyses and controlled experiments, we show that the proposed parameterization yields stable feature updates, learning rate transfer, and improved performance as width and depth increase. For SGD, we identify graph-dependent first-layer correction factors and show that their use can accelerate early training in graphs with sparse bag-of-words inputs. For Adam, we explore how different message passing normalizations affect early- and late-training transfer behavior, illustrating the importance of message passing normalization and advocating for an associated hyperparameter. For AdamW, we adapt a parameterization that allows for the joint transfer of weight decay and learning rate. Together, these results provide a practical recipe for scaling GNNs across a variety of learning tasks and training scenarios.

Problem

• hyperparameter transfer
• graph neural networks
• model scaling
• optimization

Innovation

• hyperparameter transfer
• graph neural networks
• scaling laws
• message passing normalization
• optimization