ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration

2026-08-09

AI Summary

This work addresses the lack of a unified and reproducible benchmark for evaluating collaborative processes in large language model (LLM)-based multi-agent systems, where reliance solely on outcomes or LLM-based judgments often leads to inconsistent assessments. The authors propose the first general evaluation paradigm grounded in collaboration graphs and reference forests: execution trajectories are mapped into collaboration graphs and structurally compared against task-specific reference forests, enabling efficient scoring without additional LLM inference. This approach preserves collaboration diversity and supports cross-method, reusable, and low-overhead structural evaluation. Built upon this framework, ForestBench comprises 844 tasks with ten successful reference graphs per task, delivers millisecond-level scoring, and demonstrates robust stability and validity across six mainstream multi-agent frameworks.

Abstract

Multi-agent systems (MAS) built on Large Language Models (LLMs) are proliferating rapidly, but their heterogeneous execution traces provide no common basis for evaluation across methods. Outcome-only benchmarks discard collaborations, whereas LLM-as-Judge evaluation requires additional, model-dependent inference and can vary with the LLM and rubric. We introduce a generalizable evaluation framework that maps native MAS traces into a shared space of unified collaboration graphs, enabling different methods to be evaluated under the same representation, reference set, and metric panel. Candidate graphs are compared with a query-specific reference forest. Each forest is a benchmark-provided collection of verified-success graphs: it records diverse ways in which representative MAS methods can complete the task, rather than prescribing a unique optimal process. Instantiating the framework as ForestBench, we filter $844$ collaboration-necessary queries from seven public datasets, precompute ten successful target-conditioned reference graphs per query, and evaluate six representative MAS frameworks. Controlled backbone, reference-construction, and perturbation studies test the stability and scope of evaluation. Once the benchmark forests are built, ForestBench scores a trace in milliseconds without further LLM inference, providing a reusable structural basis for comparing diverse MAS collaboration traces.

Problem

• multi-agent systems
• evaluation framework
• collaboration graphs
• benchmarking
• Large Language Models

Innovation

• collaboration graph
• multi-agent systems
• unified evaluation framework
• reference forest
• LLM-based MAS