Scalability Matters: Overcoming Challenges in InstructGLM with Similarity-Degree-Based Sampling

2025-05-02

AI Summary

Large language models (LLMs) struggle to model graph-structured data efficiently due to token-length limitations, lack of native graph awareness, and reliance on graph neural networks (GNNs). To address this, we propose InstructGLM—the first GNN-free, pure-LLM framework for graph learning. Our method introduces: (1) a similarity-degree joint-driven biased random walk mechanism for adaptive and scalable graph substructure sampling; and (2) a graph-structured instruction-tuning paradigm coupled with token-efficient serialization, overcoming the long-graph-input bottleneck. Experiments demonstrate that InstructGLM matches or surpasses state-of-the-art GNN baselines on node classification and link prediction across multiple large-scale graph benchmarks, while reducing redundant token consumption by over 30%. The framework achieves high scalability and inherent interpretability without compromising performance.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in various natural language processing tasks; however, their application to graph-related problems remains limited, primarily due to scalability constraints and the absence of dedicated mechanisms for processing graph structures. Existing approaches predominantly integrate LLMs with Graph Neural Networks (GNNs), using GNNs as feature encoders or auxiliary components. However, directly encoding graph structures within LLMs has been underexplored, particularly in the context of large-scale graphs where token limitations hinder effective representation. To address these challenges, we propose SDM-InstructGLM, a novel instruction-tuned Graph Language Model (InstructGLM) framework that enhances scalability and efficiency without relying on GNNs. Our method introduces a similarity-degree-based biased random walk mechanism, which selectively samples and encodes graph information based on node-feature similarity and degree centrality, ensuring an adaptive and structured representation within the LLM. This approach significantly improves token efficiency, mitigates information loss due to random sampling, and enhances performance on graph-based tasks such as node classification and link prediction. Furthermore, our results demonstrate the feasibility of LLM-only graph processing, enabling scalable and interpretable Graph Language Models (GLMs) optimized through instruction-based fine-tuning. This work paves the way for GNN-free approaches to graph learning, leveraging LLMs as standalone graph reasoning models. Our source code is available on GitHub.

Problem

• LLMs struggle with graph tasks due to scalability and structure limitations
• Existing methods rely on GNNs, lacking direct graph encoding in LLMs
• Token inefficiency hinders effective large-scale graph representation in LLMs

Innovation

• Similarity-degree-based biased random walk mechanism
• LLM-only graph processing without GNNs
• Instruction-tuned Graph Language Model framework