Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation

2026-06-21

AI Summary

Current language models struggle to effectively encode global graph information, limiting their capacity for graph understanding. This work proposes Graph-aware Low-Rank Adaptation (GaRA), a method that generates low-rank weight updates conditioned on graph structure to directly inject whole-graph information into the language model’s hidden representations. To stabilize training and mitigate optimization bias, GaRA incorporates norm control during adaptation. Notably, this approach is the first to enable task-specific weight generation grounded in graph topology. Extensive experiments demonstrate that GaRA significantly outperforms existing baselines on zero-shot graph learning tasks, confirming that explicit injection of global graph information substantially enhances both the reasoning capability and generalization of language models on graph-structured data.

Abstract

Graph neural networks (GNNs) tightly couple their input-output parameters to dataset-specific feature spaces and target sets, exhibiting limited transferability across different datasets. In contrast, language models (LMs) generalize flexibly via a unified input-output interface, motivating recent attempts to adapt LMs to graph tasks. However, existing methods struggle to encode whole-graph information, leading to potential information loss and suboptimal graph understanding. In this work, we propose a novel weight-level information injection paradigm for adapting LMs to graph tasks. This paradigm injects whole-graph information by generating task-specific weight updates that interact directly with hidden representations. Instantiating this paradigm following low-rank adaptation (LoRA), we introduce GaRA, a Graph-aware LoRA generation model. GaRA constructs low-rank weight updates conditioned on the original graph structures and constrains the norm of the generated updates, thus injecting whole-graph information and avoiding the optimization bias in the weight generation. Empirical studies demonstrate that GaRA consistently outperforms baselines on zero-shot graph learning tasks.

Problem

• graph tasks
• whole-graph information
• language models
• information loss
• graph understanding

Innovation

• Graph-aware LoRA
• weight-level injection
• whole-graph information
• low-rank adaptation
• zero-shot graph learning