Scalable and Accurate Graph Reasoning with LLM-based Multi-Agents

2024-10-07

arXiv.org

AI Summary

To address the low accuracy of large language models (LLMs) in complex graph reasoning—stemming from limitations in long-context processing and explicit graph-structure modeling—this paper proposes GraphAgent-Reasoner, a fine-tuning-free multi-agent framework. Grounded in distributed graph computation theory, it decomposes graph reasoning tasks in a node-centric manner and delegates them to decentralized, collaborative agents that operate in parallel. Native LLM coordination is achieved via zero-shot prompt engineering, eliminating the need for parameter updates. Evaluated on the GraphInstruct benchmark, GraphAgent-Reasoner achieves 99.8% accuracy on small graphs and maintains linear scalability on large graphs (>1000 nodes), substantially outperforming state-of-the-art closed-source and fine-tuned open-source models. The framework has been successfully deployed in real-world applications, including web page importance analysis.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent research has explored the use of Large Language Models (LLMs) for tackling complex graph reasoning tasks. However, due to the intricacies of graph structures and the inherent limitations of LLMs in handling long text, current approaches often fail to deliver satisfactory accuracy, even on small-scale graphs and simple tasks. To address these challenges, we introduce GraphAgent-Reasoner, a fine-tuning-free framework that utilizes a multi-agent collaboration strategy for explicit and precise graph reasoning. Inspired by distributed graph computation theory, our framework decomposes graph problems into smaller, node-centric tasks that are distributed among multiple agents. The agents collaborate to solve the overall problem, significantly reducing the amount of information and complexity handled by a single LLM, thus enhancing the accuracy of graph reasoning. By simply increasing the number of agents, GraphAgent-Reasoner can efficiently scale to accommodate larger graphs with over 1,000 nodes. Evaluated on the GraphInstruct dataset, our framework demonstrates near-perfect accuracy on polynomial-time graph reasoning tasks, significantly outperforming the best available models, both closed-source and fine-tuned open-source variants. Our framework also demonstrates the capability to handle real-world graph reasoning applications such as webpage importance analysis.

Problem

• Improves graph reasoning accuracy by reducing single LLM complexity
• Enables scalable processing of large graphs with 1000+ nodes
• Solves limitations in handling complex graph structures with LLMs

Innovation

• Multi-agent collaboration strategy for graph reasoning
• Decomposes graph problems into node-centric tasks
• Scales to large graphs by increasing agent count