Towards Heterogeneous Continual Graph Learning via Meta-knowledge Distillation

2025-05-23

AI Summary

To address catastrophic forgetting in continual learning on dynamically evolving heterogeneous graphs, this paper proposes a Meta-Knowledge Distillation (MKD) framework. Methodologically: (1) it introduces a novel diversity-aware sampling strategy based on meta-path structural correlation to enhance coverage of historical task representations; (2) it designs a semantic-level meta-path attention distillation module that enables cross-task knowledge transfer via attention distribution alignment; and (3) it integrates meta-learning with heterogeneous graph neural networks to improve rapid adaptation to newly emerging nodes and relations. Evaluated on three benchmark datasets, MKD consistently outperforms state-of-the-art methods, achieving a 5.2% improvement in average accuracy across old and new tasks and reducing forgetting rate by 37.6%. The framework thus delivers both substantial performance gains and enhanced training stability.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Machine learning on heterogeneous graphs has experienced rapid advancement in recent years, driven by the inherently heterogeneous nature of real-world data. However, existing studies typically assume the graphs to be static, while real-world graphs are continuously expanding. This dynamic nature requires models to adapt to new data while preserving existing knowledge. To this end, this work addresses the challenge of continual learning on heterogeneous graphs by introducing the Meta-learning based Knowledge Distillation framework (MKD), designed to mitigate catastrophic forgetting in evolving heterogeneous graph structures. MKD combines rapid task adaptation through meta-learning on limited samples with knowledge distillation to achieve an optimal balance between incorporating new information and maintaining existing knowledge. To improve the efficiency and effectiveness of sample selection, MKD incorporates a novel sampling strategy that selects a small number of target-type nodes based on node diversity and maintains fixed-size buffers for other types. The strategy retrieves first-order neighbors along metapaths and selects important neighbors based on their structural relevance, enabling the sampled subgraphs to retain key topological and semantic information. In addition, MKD introduces a semantic-level distillation module that aligns the attention distributions over different metapaths between teacher and student models, encouraging semantic consistency beyond the logit level. Comprehensive evaluations across three benchmark datasets validate MKD's effectiveness in handling continual learning scenarios on expanding heterogeneous graphs.

Problem

• Addresses continual learning on dynamic heterogeneous graphs
• Mitigates catastrophic forgetting in evolving graph structures
• Balances new information integration with existing knowledge preservation

Innovation

• Meta-learning based Knowledge Distillation framework
• Novel sampling strategy for node diversity
• Semantic-level distillation module for consistency