Deep Semantic Graph Learning via LLM based Node Enhancement

2025-02-11

AI Summary

Existing graph neural networks (GNNs) suffer from shallow semantic representation of node textual features, limiting their ability to capture deep linguistic and contextual semantics. Method: We propose LLM-GraphTransformer, an end-to-end fusion framework that first leverages large language models (LLMs) to generate high-density, semantically rich text embeddings for nodes—replacing conventional handcrafted or shallow encoders—and then employs a dedicated Graph Transformer module to jointly model local neighborhood structures and global topological dependencies via multi-head self-attention. Contribution/Results: This work pioneers the tight integration of LLM-based semantic enhancement with Graph Transformer architecture, overcoming the inherent limitation of GNNs in shallow textual feature modeling. Extensive experiments on multiple Chinese and English benchmark graph datasets for node classification demonstrate consistent and significant accuracy improvements, validating the generalizability and effectiveness of LLM-enhanced features in graph representation learning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Graph learning has attracted significant attention due to its widespread real-world applications. Current mainstream approaches rely on text node features and obtain initial node embeddings through shallow embedding learning using GNNs, which shows limitations in capturing deep textual semantics. Recent advances in Large Language Models (LLMs) have demonstrated superior capabilities in understanding text semantics, transforming traditional text feature processing. This paper proposes a novel framework that combines Graph Transformer architecture with LLM-enhanced node features. Specifically, we leverage LLMs to generate rich semantic representations of text nodes, which are then processed by a multi-head self-attention mechanism in the Graph Transformer to capture both local and global graph structural information. Our model utilizes the Transformer's attention mechanism to dynamically aggregate neighborhood information while preserving the semantic richness provided by LLM embeddings. Experimental results demonstrate that the LLM-enhanced node features significantly improve the performance of graph learning models on node classification tasks. This approach shows promising results across multiple graph learning tasks, offering a practical direction for combining graph networks with language models.

Problem

• Enhancing graph node semantics
• Integrating LLMs for deep learning
• Improving node classification accuracy

Innovation

• LLM-enhanced node features
• Graph Transformer architecture
• dynamic neighborhood aggregation