MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability

2026-01-01

arXiv.org

AI Summary

This work addresses the absence of standardized evaluation methodologies for large language model–based multi-agent systems (MAS) in terms of testing, reliability, and observability. We propose a unified execution framework that enables seamless integration of both native and third-party MAS through lightweight adapters, standardized interfaces, and a cross-framework example library, while capturing framework-agnostic execution traces and system-level metrics such as latency, cost, and failure rates. Our systematic evaluation—the first of its kind—demonstrates that MAS architectural design predominantly governs runtime stability, performance variability, and the trade-offs among cost, latency, and accuracy, with effects far outweighing those of backend model choices or tool configurations. These findings are empirically validated across twelve representative MAS implementations.

Technology Category

• Multiagent Systems: Multiagent Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

We present MAESTRO, an evaluation suite for the testing, reliability, and observability of LLM-based MAS. MAESTRO standardizes MAS configuration and execution through a unified interface, supports integrating both native and third-party MAS via a repository of examples and lightweight adapters, and exports framework-agnostic execution traces together with system-level signals (e.g., latency, cost, and failures). We instantiate MAESTRO with 12 representative MAS spanning popular agentic frameworks and interaction patterns, and conduct controlled experiments across repeated runs, backend models, and tool configurations. Our case studies show that MAS executions can be structurally stable yet temporally variable, leading to substantial run-to-run variance in performance and reliability. We further find that MAS architecture is the dominant driver of resource profiles, reproducibility, and cost-latency-accuracy trade-off, often outweighing changes in backend models or tool settings. Overall, MAESTRO enables systematic evaluation and provides empirical guidance for designing and optimizing agentic systems.

Problem

• Multi-Agent Systems
• LLM-based MAS
• Evaluation Suite
• Reliability
• Observability

Innovation

• Multi-Agent Systems
• LLM Evaluation
• Reliability
• Observability
• Execution Tracing