AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

2026-05-08

AI Summary

This study addresses the limitations of outcome-based evaluation in detecting multi-hop procedural failures in multi-agent collaboration, which often arise from constraint loss or error propagation. To this end, the authors construct a human-validated benchmark comprising 900 tasks spanning software engineering, DevOps, and data engineering, enabling the first systematic identification and quantification of four classes of latent behavioral risks. Their analysis reveals that communication topology is a critical determinant of reliable information transmission—architectural design cannot be substituted by increased model capability alone. Through multi-agent simulations and DAG-based topological analysis, evaluated with mainstream large language models including GPT-4.1 mini, they find significant performance disparities across models under different risk conditions. Notably, convergence nodes introduce synthesis bottlenecks that exacerbate constraint loss, with communication topology accounting for 7%–40% of the variance in information survival rates.

Abstract

Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint, the system's final output may look correct even though the reasoning chain was quietly corrupted, and existing outcome-based evaluations are blind to such multi-hop process failures. To make these vulnerabilities measurable before deployment, we introduce AgentCollabBench, a diagnostic benchmark of 900 human-validated tasks spanning software engineering, DevOps, and data engineering. Each task isolates one of four behavioral risks: instruction decay (does a constraint survive peer pressure?), false-belief contagion (does a falsehood spread through consensus?), context leakage (does information bleed between tasks?), and tracer durability (does marked data reach the final agent?). Evaluating four modern LLMs (GPT 4.1 mini, Gemini 2.5 Flash Lite, Qwen-3.5-35B-A3B, and Llama 3.1 8B Instruct), we expose model-specific vulnerability profiles invisible to outcome-only evaluation; Qwen-3.5-35B-A3B, for example, leads on tracer durability and instruction stability, while GPT 4.1 mini leads on leakage containment and false-belief resistance. Beyond per-model differences, communication topology emerges as a primary risk factor that explains 7-40% of the variance in multi-hop information survival. The effect traces to a synthesis bottleneck specific to converging-DAG nodes: an agent weighing competing parent inputs discards constraints carried by a minority branch, a bottleneck structurally absent from linear chains. AgentCollabBench demonstrates that suboptimal topology can silently erase the safeguards of highly capable models, arguing that multi-agent reliability is fundamentally a structural problem and that scaling model intelligence alone is no substitute for architecture.

Problem

• multi-agent collaboration
• process failure
• communication topology
• constraint preservation
• information leakage

Innovation

• multi-agent collaboration
• diagnostic benchmark
• communication topology
• instruction decay
• information bottleneck