Towards Graph Foundation Models: Training on Knowledge Graphs Enables Transferability to General Graphs

2024-10-16

AI Summary

Current graph foundation models exhibit limited generalization capability and require task-specific fine-tuning for cross-graph applications. To address this, we propose SCR, the first knowledge graph (KG)-based graph foundation model enabling unified pretraining and zero-shot inductive reasoning across node-, link-, and graph-level tasks. Our method introduces two key innovations: (1) a semantic-conditioned message-passing mechanism that jointly encodes structural topology and semantic relations—overcoming the traditional decoupling of structure and semantics in KG reasoning; and (2) a task-adaptive unified KG topology modeling framework that enhances representation generality. Evaluated on 38 heterogeneous graph datasets, SCR significantly outperforms existing foundation models and supervised baselines, demonstrating strong zero-shot generalization and robust transferability across diverse tasks and domains.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Other Foundations of Knowledge Representation & Reasoning
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Inspired by the success of large language models, there is a trend toward developing graph foundation models to conduct diverse downstream tasks in various domains. However, current models often require extra fine-tuning to apply their learned structural and semantic representations to new graphs, which limits their versatility. Recent breakthroughs in zero-shot inductive reasoning on knowledge graphs (KGs), offer us a new perspective on extending KG reasoning to general graph applications. In this paper, we introduce SCR, a unified graph reasoning framework designed to train on knowledge graphs and effectively generalize across a wide range of graph tasks and domains. We begin by designing the task-specific KG structures to establish a unified topology for different task formats. Then we propose semantic-conditioned message passing, a novel mechanism addressing the inherent semantic isolation in traditional KG reasoning, by jointly modeling structural and semantic invariance patterns in graph representations. To demonstrate the effectiveness, we evaluate the inductive reasoning capability of SCR using 38 diverse graph datasets, covering node-level, link-level, and graph-level tasks across multiple domains. Our results show substantial performance gains over existing foundation models and supervised baselines, highlighting the efficacy and adaptability of our approach.

Problem

• Develop graph foundation models for diverse tasks
• Overcome limitations in transferring learned representations to new graphs
• Extend knowledge graph reasoning to general graph applications

Innovation

• Unified graph reasoning framework for diverse tasks
• Semantic-conditioned message passing mechanism
• Training on knowledge graphs for generalizability