TaLK: Text-attributed Graph Dataset Distillation via Coupling Language Model with Graph-Aware Kernel

2026-06-22

AI Summary

This work addresses the high computational cost of existing textual attributed graph (TAG) learning methods, which rely on joint training of language models and graph neural networks, and the inability of conventional dataset distillation techniques to effectively preserve multimodal characteristics. To overcome these limitations, the authors propose TaLK, a novel approach that, for the first time, couples graph-aware neural tangent kernels with pretrained language models to enable efficient TAG dataset distillation. TaLK preserves both textual semantics and graph structural information without requiring repeated full-model retraining. Experimental results demonstrate that TaLK achieves 97% of the performance obtained using the full dataset while using only 1% of synthesized data across multiple TAG benchmarks, significantly outperforming existing baselines and substantially improving both distillation efficiency and multimodal fidelity.

Abstract

Text-attributed graphs (TAGs) are widely used in many real-world domains, and learning on TAGs requires jointly modeling text semantics and graph structure. A standard approach for modeling TAGs is to combine a language model (LM) and a graph neural network (GNN), but joint training is computationally expensive and difficult to scale. Dataset distillation is a promising way to reduce training costs, but existing methods are not well suited to TAGs because they are typically designed for a single modality or still require repeatedly training expensive LM-GNN models on the full dataset during distillation. To address this, we propose TaLK, an effective dataset distillation method for TAGs that couples an LM with a graph-aware neural tangent kernel.This design enables efficient dataset distillation, avoiding repeated joint training on the full dataset while reflecting both textual and structural information for effective TAG learning.Experiments on multiple TAG benchmarks show that TaLK consistently outperforms existing baselines and achieves up to 97% of full-dataset performance with only 1% synthetic data.

Problem

• Text-attributed graphs
• Dataset distillation
• Language model
• Graph neural network
• Multimodal learning

Innovation

• Text-attributed Graphs
• Dataset Distillation
• Language Model
• Graph-aware Kernel
• Neural Tangent Kernel