GriNNder: Breaking the Memory Capacity Wall in Full-Graph GNN Training with Storage Offloading

2026-05-12

AI Summary

This work addresses the challenge of full-graph GNN training, which is severely constrained by GPU and host memory capacities, limiting scalability to large-scale graphs. To overcome this bottleneck, the study introduces storage offloading into full-graph GNN training for the first time, establishing a three-tier hierarchy comprising GPU, host memory, and NVMe SSDs. The authors propose a structured storage offloading framework that integrates partitioned caching, gradient reaggregation, and lightweight graph partitioning, effectively aligning with GNNs’ access patterns and dependency characteristics. Evaluated across diverse models and datasets, the approach achieves up to a 9.78× speedup over the state-of-the-art while enabling a single GPU to match the throughput of distributed systems, thereby making previously infeasible large-scale full-graph training tractable.

Abstract

Full-graph training of graph neural networks (GNNs) is widely used as it enables direct validation of algorithmic improvements by preserving complete neighborhood information. However, it typically requires multiple GPUs or servers, incurring substantial hardware and inter-device communication costs. While existing single-server methods reduce infrastructure requirements, they remain constrained by GPU and host memory capacity as graph sizes increase. To address this limitation, we introduce GriNNder, which is the first work to leverage storage devices to enable full-graph training even with limited memory. Because modern NVMe SSDs offer multi-terabyte capacities and bandwidths exceeding 10 GB/s, they provide an appealing option when memory resources are scarce. Yet, directly applying storage-based methods from other domains fails to address the unique access patterns and data dependencies in full-graph GNN training. GriNNder tackles these challenges by structured storage offloading (SSO), a framework that manages the GPU-host-storage hierarchy through coordinated cache, (re)gather, and bypass mechanisms. To realize the framework, we devise (i) a partition-wise caching strategy for host memory that exploits the observation on cross-partition dependencies, (ii) a regathering strategy for gradient computation that eliminates redundant storage operations, and (iii) a lightweight partitioning scheme that mitigates the memory requirements of existing graph partitioners. In experiments performed over various models and datasets, GriNNder achieves up to 9.78x speedup over state-of-the-art baselines and throughput comparable to distributed systems, enabling previously infeasible large-scale full-graph training even on a single GPU.

Problem

• full-graph GNN training
• memory capacity
• storage offloading
• large-scale graphs
• single-GPU training

Innovation

• storage offloading
• full-graph GNN training
• structured storage offloading
• memory hierarchy optimization
• NVMe SSD