GOFA: A Generative One-For-All Model for Joint Graph Language Modeling

2024-07-12

arXiv.org

AI Summary

Existing graph foundation models (GFMs) struggle to simultaneously achieve strong task generalization and high-fidelity graph structure modeling. Method: We propose the first unified GFM framework supporting self-supervised pretraining, flexible task generalization, and explicit graph-structure awareness. Our approach innovatively integrates learnable graph neural network (GNN) layers into a frozen large language model (LLM), forming an end-to-end generative architecture. We introduce, for the first time, a joint pretraining paradigm comprising graph-level next-token prediction, graph question answering, and structural reconstruction. Contribution/Results: This design enables organic synergy between semantic understanding and topological modeling, significantly enhancing zero-shot graph-structure awareness and contextual reasoning. Extensive experiments demonstrate consistent and substantial improvements over both graph language models and GNN-based baselines across diverse downstream tasks. The code is publicly available.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Foundation models, such as Large Language Models (LLMs) or Large Vision Models (LVMs), have emerged as one of the most powerful tools in the respective fields. However, unlike text and image data, graph data do not have a definitive structure, posing great challenges to developing a Graph Foundation Model (GFM). For example, current attempts at designing general graph models either transform graph data into a language format for LLM-based prediction or still train a GNN model with LLM as an assistant. The former can handle unlimited tasks, while the latter captures graph structure much better -- yet, no existing work can achieve both simultaneously. In this paper, we identify three key desirable properties of a GFM: self-supervised pretraining, fluidity in tasks, and graph awareness. To account for these properties, we extend the conventional language modeling to the graph domain and propose a novel generative graph language model GOFA to solve the problem. The model interleaves randomly initialized GNN layers into a frozen pre-trained LLM so that the semantic and structural modeling abilities are organically combined. GOFA is pre-trained on newly proposed graph-level next-word prediction, question-answering, and structural tasks to obtain the above GFM properties. The pre-trained model is further fine-tuned on downstream tasks to obtain task-solving ability. The fine-tuned model is evaluated on various downstream tasks, demonstrating a strong ability to solve structural and contextual problems in zero-shot scenarios. The code is available at https://github.com/JiaruiFeng/GOFA.

Problem

• Developing a Graph Foundation Model (GFM) for diverse graph tasks
• Combining semantic and structural modeling in graph-language integration
• Achieving self-supervised pretraining and graph awareness simultaneously

Innovation

• Interleaves GNN layers into frozen LLM
• Pre-trained on graph-level multi-tasks
• Combines semantic and structural modeling