Computation-friendly Graph Neural Network Design by Accumulating Knowledge on Large Language Models

2024-08-13

arXiv.org

AI Summary

Existing large language models (LLMs) underperform in data-sensitive graph neural network (GNN) architecture design, primarily due to their inability to model the complex mapping between graph structural properties and architectural performance, and their susceptibility to misleading inputs. Method: We propose a knowledge retrieval and driven search framework—the first to systematically inject structured GNN design expertise into LLMs. It comprises a knowledge retrieval pipeline and a structured experience encoding module, integrated with prompt engineering and lightweight Bayesian optimization to enable efficient exploration-exploitation trade-offs. Contribution/Results: Our method requires no pretraining and generates an initial architecture achieving top-5.77% performance on unseen datasets within seconds; only a few iterations suffice to reach state-of-the-art (SOTA) performance. It significantly reduces design overhead, overcoming the inefficiencies of traditional trial-and-error and slow meta-knowledge accumulation.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Graph Neural Networks (GNNs), like other neural networks, have shown remarkable success but are hampered by the complexity of their architecture designs, which heavily depend on specific data and tasks. Traditionally, designing proper architectures involves trial and error, which requires intensive manual effort to optimize various components. To reduce human workload, researchers try to develop automated algorithms to design GNNs. However, both experts and automated algorithms suffer from two major issues in designing GNNs: 1) the substantial computational resources expended in repeatedly trying candidate GNN architectures until a feasible design is achieved, and 2) the intricate and prolonged processes required for humans or algorithms to accumulate knowledge of the interrelationship between graphs, GNNs, and performance. To further enhance the automation of GNN architecture design, we propose a computation-friendly way to empower Large Language Models (LLMs) with specialized knowledge in designing GNNs, thereby drastically shortening the computational overhead and development cycle of designing GNN architectures. Our framework begins by establishing a knowledge retrieval pipeline that comprehends the intercorrelations between graphs, GNNs, and performance. This pipeline converts past model design experiences into structured knowledge for LLM reference, allowing it to quickly suggest initial model proposals. Subsequently, we introduce a knowledge-driven search strategy that emulates the exploration-exploitation process of human experts, enabling quick refinement of initial proposals within a promising scope. Extensive experiments demonstrate that our framework can efficiently deliver promising (e.g., Top-5.77%) initial model proposals for unseen datasets within seconds and without any prior training and achieve outstanding search performance in a few iterations.

Problem

• LLMs struggle with specialized GNN design tasks
• Difficulty in modeling graph-architecture relationships and handling noise
• Challenges in accumulating and applying data-specific design knowledge

Innovation

• Converts design experiences into structured knowledge priors
• Aligns empirical filtering with adaptive literature insights
• Constructs meta-knowledge for unseen graph understanding