Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference

2024-11-21

arXiv.org

AI Summary

To address the high inference latency and poor deployability of heterogeneous graph neural networks (HGNNs) in latency-sensitive applications—stemming from their structural dependencies—this paper proposes HG2M and HG2M+, the first frameworks to distill HGNN knowledge into graph-structure-free MLP student models. We introduce two novel distillation mechanisms: reliable node distillation and reliable meta-path distillation, enabling the MLP to accurately capture heterogeneous semantics without accessing the original graph structure. Evaluated on six standard benchmarks, our models match or surpass state-of-the-art HGNNs in accuracy while significantly outperforming baseline MLPs. On the large-scale IGB-3M-19 dataset, HG2M achieves a 379.24× inference speedup, delivering millisecond-level latency suitable for real-time edge deployment. This work establishes the first unified solution achieving HGNN-level predictive performance with ultra-low inference latency.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Heterogeneous Graph Neural Networks (HGNNs) have achieved promising results in various heterogeneous graph learning tasks, owing to their superiority in capturing the intricate relationships and diverse relational semantics inherent in heterogeneous graph structures. However, the neighborhood-fetching latency incurred by structure dependency in HGNNs makes it challenging to deploy for latency-constrained applications that require fast inference. Inspired by recent GNN-to-MLP knowledge distillation frameworks, we introduce HG2M and HG2M+ to combine both HGNN's superior performance and MLP's efficient inference. HG2M directly trains student MLPs with node features as input and soft labels from teacher HGNNs as targets, and HG2M+ further distills reliable and heterogeneous semantic knowledge into student MLPs through reliable node distillation and reliable meta-path distillation. Experiments conducted on six heterogeneous graph datasets show that despite lacking structural dependencies, HG2Ms can still achieve competitive or even better performance than HGNNs and significantly outperform vanilla MLPs. Moreover, HG2Ms demonstrate a 379.24$ imes$ speedup in inference over HGNNs on the large-scale IGB-3M-19 dataset, showcasing their ability for latency-sensitive deployments.

Problem

• Reduce inference latency in heterogeneous graph learning tasks
• Combine HGNN performance with MLP efficiency via distillation
• Enable fast inference for latency-sensitive applications

Innovation

• HG2M distills HGNN knowledge into MLPs
• HG2M+ adds reliable semantic knowledge distillation
• Achieves fast inference with competitive accuracy