Are Large Language Models In-Context Graph Learners?

2025-02-19

AI Summary

Large language models (LLMs) inherently lack native support for non-Euclidean graph structures, leading to substantially inferior performance on graph learning tasks compared to graph neural networks (GNNs). Method: We propose a zero-shot, fine-tuning-free graph learning paradigm that reframes graph learning as a retrieval-augmented generation (RAG) task: nodes or edges serve as queries, the entire graph as structured context, and graph-structure-aware prompting—combined with context-aware instance instantiation—guides LLM reasoning without modifying model weights. Contribution/Results: Our approach overcomes fundamental limitations of LLMs in graph representation learning. Evaluated on multiple standard graph benchmarks, it achieves significant performance gains, matching or even surpassing lightweight GNNs under API-only access or frozen-weight settings. This demonstrates, for the first time, the feasibility and potential of LLMs as general-purpose graph learners—without architectural modification or parameter updates.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have demonstrated remarkable in-context reasoning capabilities across a wide range of tasks, particularly with unstructured inputs such as language or images. However, LLMs struggle to handle structured data, such as graphs, due to their lack of understanding of non-Euclidean structures. As a result, without additional fine-tuning, their performance significantly lags behind that of graph neural networks (GNNs) in graph learning tasks. In this paper, we show that learning on graph data can be conceptualized as a retrieval-augmented generation (RAG) process, where specific instances (e.g., nodes or edges) act as queries, and the graph itself serves as the retrieved context. Building on this insight, we propose a series of RAG frameworks to enhance the in-context learning capabilities of LLMs for graph learning tasks. Comprehensive evaluations demonstrate that our proposed RAG frameworks significantly improve LLM performance on graph-based tasks, particularly in scenarios where a pretrained LLM must be used without modification or accessed via an API.

Problem

• LLMs struggle with graph data
• Propose RAG frameworks for graphs
• Enhance LLM performance on graph tasks

Innovation

• LLMs enhance graph learning via RAG.
• Graph context serves as retrieval source.
• RAG frameworks boost LLM graph performance.