The Collaboration Gap

2025-11-04

AI Summary

This study addresses collaborative failure among heterogeneous AI agents in partially observable multi-agent systems. To overcome the lack of large-scale empirical evaluation, we introduce a tunably complex maze-solving benchmark and systematically evaluate 32 state-of-the-art models under solo, homogeneous, and heterogeneous pairing settings. We empirically identify and quantify the “collaboration gap”—a significant performance degradation exhibited by high-capability agents when collaborating with weaker counterparts. To mitigate this gap, we propose relay reasoning: a novel paradigm wherein a stronger agent performs high-level planning while a weaker agent executes low-level actions. Experiments demonstrate that relay reasoning substantially narrows the collaboration gap, boosting success rates for distilled small-scale model pairs by several-fold. Our work advances collaboration-aware evaluation and training paradigms, offering broadly applicable insights for both AI–AI and human–AI coordination.

Technology Category

• Multiagent Systems: Coordination and Collaboration
• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

The trajectory of AI development suggests that we will increasingly rely on agent-based systems composed of independently developed agents with different information, privileges, and tools. The success of these systems will critically depend on effective collaboration among these heterogeneous agents, even under partial observability. Despite intense interest, few empirical studies have evaluated such agent-agent collaboration at scale. We propose a collaborative maze-solving benchmark that (i) isolates collaborative capabilities, (ii) modulates problem complexity, (iii) enables scalable automated grading, and (iv) imposes no output-format constraints, preserving ecological plausibility. Using this framework, we evaluate 32 leading open- and closed-source models in solo, homogeneous, and heterogeneous pairings. Our results reveal a"collaboration gap": models that perform well solo often degrade substantially when required to collaborate. Collaboration can break down dramatically; for instance, small distilled models that solve mazes well alone may fail almost completely in certain pairings. We find that starting with the stronger agent often improves outcomes, motivating a"relay inference"approach where the stronger agent leads before handing off to the weaker one, closing much of the gap. Our findings argue for (1) collaboration-aware evaluation, (2) training strategies developed to enhance collaborative capabilities, and (3) interaction design that reliably elicits agents'latent skills, guidance that applies to AI-AI and human-AI collaboration.

Problem

• Evaluating agent-agent collaboration at scale in AI systems
• Addressing performance degradation when AI models collaborate
• Developing benchmarks to isolate and measure collaborative capabilities

Innovation

• Proposed collaborative maze-solving benchmark for evaluation
• Evaluated 32 models in solo and paired settings
• Introduced relay inference approach to improve collaboration