A Comprehensive Analysis on LLM-based Node Classification Algorithms

2025-02-02

AI Summary

The application of large language models (LLMs) to graph node classification lacks systematic evaluation and practical guidance. Method: We introduce LLMNodeBed, the first open-source benchmark platform for LLM-based node classification, encompassing 10 diverse graph datasets, 8 representative LLM methods, and three learning paradigms—prompt engineering, fine-tuning, and graph-augmented embedding—supported by over 2,200 experiments. Contribution/Results: We establish the first fair comparison framework for LLMs in node classification and derive eight empirically grounded insights: (i) LLMs significantly outperform traditional GNNs in semi-supervised settings; (ii) zero-shot graph foundation models remain inferior to GPT-4o; (iii) multi-scale open-weight models (e.g., Llama, Qwen) enable standardized benchmarking against proprietary strong models; and (iv) LLMs achieve state-of-the-art performance under semi-supervised learning. The platform is fully open-sourced—including code, data, and configurations—to ensure reproducibility and advance rigorous, comparable research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Node classification is a fundamental task in graph analysis, with broad applications across various fields. Recent breakthroughs in Large Language Models (LLMs) have enabled LLM-based approaches for this task. Although many studies demonstrate the impressive performance of LLM-based methods, the lack of clear design guidelines may hinder their practical application. In this work, we aim to establish such guidelines through a fair and systematic comparison of these algorithms. As a first step, we developed LLMNodeBed, a comprehensive codebase and testbed for node classification using LLMs. It includes ten datasets, eight LLM-based algorithms, and three learning paradigms, and is designed for easy extension with new methods and datasets. Subsequently, we conducted extensive experiments, training and evaluating over 2,200 models, to determine the key settings (e.g., learning paradigms and homophily) and components (e.g., model size) that affect performance. Our findings uncover eight insights, e.g., (1) LLM-based methods can significantly outperform traditional methods in a semi-supervised setting, while the advantage is marginal in a supervised setting; (2) Graph Foundation Models can beat open-source LLMs but still fall short of strong LLMs like GPT-4o in a zero-shot setting. We hope that the release of LLMNodeBed, along with our insights, will facilitate reproducible research and inspire future studies in this field. Codes and datasets are released at href{https://llmnodebed.github.io/}{https://llmnodebed.github.io/}.

Problem

• Large Language Models
• Node Classification
• Graph Analysis

Innovation

• LLMNodeBed
• Node Classification
• Large Language Models