GLIP: Graph and LLM Joint Pretraining for Graph-Level Tasks

2026-06-29

AI Summary

This work addresses the underexplored potential of large language models (LLMs) in graph-level tasks, where existing graph pretraining approaches predominantly focus on graph neural networks (GNNs) and suffer from label scarcity. To bridge this gap, we propose GLIP, the first framework to integrate LLMs into graph-level pretraining. GLIP constructs positive and negative sample pairs via graph augmentation and employs a multi-token selection strategy coupled with a diffusion projector to jointly capture local and global structural information. The framework further introduces a unified optimization objective that combines LLM-driven semantic reasoning with contrastive alignment loss. With only minimal labeled data for fine-tuning, GLIP substantially outperforms state-of-the-art methods across multiple graph-level classification and reasoning benchmarks, demonstrating the effectiveness and superiority of jointly pretraining graph structures with LLMs.

Abstract

Graphs are widely used to model relational systems, with applications in domains such as social networks, finance, and biomedicine. Graph neural networks (GNNs) have become a mainstream approach for learning graph representations. With the rise of large language models (LLMs), recent studies have attempted to combine GNNs with LLMs. However, most existing works concentrate on node-level and edge-level tasks, while graph-level tasks, which require capturing more complex structural and feature information, remain relatively underexplored. Moreover, graph pretraining is a widely adopted strategy to alleviate the challenge of label scarcity. Most existing approaches are designed solely for GNNs such as GraphCL, leaving LLMs uninvolved in the process. To address these limitations, we propose GLIP, a Graph-LLM JoInt Pretraining framework for graph-level tasks. GLIP first performs graph augmentation to construct positive and negative pairs and introduces a multi-token selection strategy to identify patches informative in both structure and features. It further leverages a diffusion-based projector to enrich them with contextual information, enabling GLIP to capture signals from both global and local perspectives. Finally, GLIP employs a joint objective that integrates the LLM's semantic judgments with a contrastive alignment loss, ensuring consistent supervision at both the semantic and structural levels. After pretraining, GLIP is fine-tuned with limited labeled data for downstream tasks, and extensive experiments show that it outperforms state-of-the-art methods on graph-level classification and reasoning tasks. Our source code is publicly available at https://anonymous.4open.science/r/GLIP.

Problem

• graph-level tasks
• graph pretraining
• large language models
• graph neural networks
• label scarcity

Innovation

• Graph-LLM joint pretraining
• graph-level tasks
• multi-token selection
• diffusion-based projector
• contrastive alignment