GraphEval36K: Benchmarking Coding and Reasoning Capabilities of Large Language Models on Graph Datasets

2024-06-23

AI Summary

Large language models (LLMs) exhibit significant limitations in understanding, programming, and reasoning over graph-structured data. To address this, we introduce GraphEval36K—the first large-scale, comprehensive benchmark for graph-domain evaluation—comprising 40 categories of graph programming tasks and 36,900 rigorously curated test cases, underpinned by a fine-grained taxonomy of graph tasks. We propose Structured Symbolic Decomposition (SSD), an instruction-enhancement method that integrates symbolic task decomposition with instruction tuning. We conduct systematic, cross-model evaluations across ten state-of-the-art LLMs and diverse graph types (directed/undirected, topological/algorithmic/network). Results demonstrate substantial performance gains: average pass rates increase by 8.38% (GPT-4), 6.78% (GPT-4o), 29.28% (Gemini-Pro), and 25.28% (Claude-3-Sonnet). Furthermore, our analysis reveals a rapid convergence in capability between proprietary and open-source LLMs on graph reasoning tasks.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have achieved remarkable success in natural language processing (NLP), demonstrating significant capabilities in processing and understanding text data. However, recent studies have identified limitations in LLMs' ability to manipulate, program, and reason about structured data, especially graphs. We introduce GraphEval36K, the first comprehensive graph dataset, comprising 40 graph coding problems and 36,900 test cases to evaluate the ability of LLMs on graph problem-solving. Our dataset is categorized into eight primary and four sub-categories to ensure a thorough evaluation across different types of graphs. We benchmark ten LLMs, finding that private models outperform open-source ones, though the gap is narrowing. We also analyze the performance of LLMs across directed vs undirected graphs, different kinds of graph concepts, and network models. Furthermore, to improve the usability of our evaluation framework, we propose Structured Symbolic Decomposition (SSD), an instruction-based method designed to enhance LLM performance on complex graph tasks. Results show that SSD improves the average passing rate of GPT-4, GPT-4o, Gemini-Pro and Claude-3-Sonnet by 8.38%, 6.78%, 29.28% and 25.28%, respectively.

Problem

• Evaluating LLMs on graph problem-solving
• Introducing GraphEval36K for comprehensive testing
• Proposing SSD to enhance LLM graph performance

Innovation

• GraphEval36K benchmarks graph problem-solving
• Structured Symbolic Decomposition enhances performance
• Comprehensive dataset with 40 coding problems