WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

2026-02-20

AI Summary

Current evaluations of large language model workflows lack proper calibration and fail to adequately reflect the severity of degradation. This work proposes WorkflowPerturb—the first benchmark framework based on controlled perturbations and severity grading—to systematically assess the sensitivity and calibration of multi-agent workflow metrics. By applying three types of perturbations—omission, compression, and description alteration—to 4,973 gold-standard workflows, we generate 44,757 perturbed variants. Through structured perturbation generation, multi-level perturbation control, and residual analysis of evaluation metrics, our approach reveals systematic differences among metric families, providing an interpretable and calibratable foundation for workflow evaluation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Multiagent Systems: Teamwork
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

LLM-based systems increasingly generate structured workflows for complex tasks. In practice, automatic evaluation of these workflows is difficult, because metric scores are often not calibrated, and score changes do not directly communicate the severity of workflow degradation. We introduce WorkflowPerturb, a controlled benchmark for studying workflow evaluation metrics. It works by applying realistic, controlled perturbations to golden workflows. WorkflowPerturb contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, and Description Changes), each applied at severity levels of 10%, 30%, and 50%. We benchmark multiple metric families and analyze their sensitivity and calibration using expected score trajectories and residuals. Our results characterize systematic differences across metric families and support severity-aware interpretation of workflow evaluation scores. Our dataset will be released upon acceptance.

Problem

• workflow evaluation
• metric calibration
• LLM-based systems
• workflow degradation
• automatic evaluation

Innovation

• WorkflowPerturb
• calibrated evaluation
• workflow perturbation
• multi-agent workflows
• LLM-based systems