Unifews: Unified Entry-Wise Sparsification for Efficient Graph Neural Network

2024-03-20

arXiv.org

AI Summary

Graph Neural Networks (GNNs) incur prohibitive computational overhead due to dense graph matrix operations. To address this, we propose the first end-to-end unified element-wise sparsification framework that jointly optimizes matrix entries in both graph propagation and weight transformation, enabling adaptive, hierarchical, and incremental co-pruning of edges and weights. Our method comprises three core components: (i) importance-guided joint sparsification, (ii) dynamic hierarchical scheduling, and (iii) matrix operation skipping. Furthermore, we establish the first theoretical graph optimization analysis framework tailored for sparse GNNs, providing rigorous guarantees on bounded approximation error under sparsity constraints. Evaluated on billion-edge graphs, our approach achieves up to 100× speedup in graph propagation, reduces matrix operations by 10–20×, and attains over 90% joint edge-weight pruning rate—while maintaining or surpassing baseline accuracy.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Reasoning under Uncertainty: Graphical Models
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Graph Neural Networks (GNNs) have shown promising performance in various graph learning tasks, but at the cost of resource-intensive computations. The primary overhead of GNN update stems from graph propagation and weight transformation, both involving operations on graph-scale matrices. Previous studies attempt to reduce the computational budget by leveraging graph-level or network-level sparsification techniques, resulting in downsized graph or weights. In this work, we propose Unifews, which unifies the two operations in an entry-wise manner considering individual matrix elements, and conducts joint edge-weight sparsification to enhance learning efficiency. The entry-wise design of Unifews enables adaptive compression across GNN layers with progressively increased sparsity, and is applicable to a variety of architectural designs with on-the-fly operation simplification. Theoretically, we establish a novel framework to characterize sparsified GNN learning in view of a graph optimization process, and prove that Unifews effectively approximates the learning objective with bounded error and reduced computational load. We conduct extensive experiments to evaluate the performance of our method in diverse settings. Unifews is advantageous in jointly removing more than 90% of edges and weight entries with comparable or better accuracy than baseline models. The sparsification offers remarkable efficiency improvements including 10-20x matrix operation reduction and up to 100x acceleration in graph propagation time for the largest graph at the billion-edge scale.

Problem

• Reducing computational overhead in Graph Neural Networks
• Unifying graph and weight matrix operations efficiently
• Enhancing GNN learning with adaptive compression techniques

Innovation

• Joint sparsification for graph and weight matrices
• Adaptive compression with progressive sparsity increase
• Theoretical framework for bounded-error sparsified learning