MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments

2026-06-30

AI Summary

This work addresses the lack of systematic evaluation of multimodal large language models (MLLMs) in embodied collaborative settings. To this end, we introduce MECoBench, a benchmark encompassing diverse real-world tasks, two collaboration structures, and three collaboration modes. We further develop an MLLM-based agent collaboration framework, a multimodal communication mechanism, and an embodied simulation platform. Our experiments provide the first quantitative analysis of how collaboration impacts task completion, revealing a trade-off between collaborative benefits and coordination complexity, and demonstrating enhanced robustness under noisy and exploratory conditions. Results show that collaboration consistently improves task success rates, with communication emerging as a critical factor, and the optimal collaboration mode depending on team size and model capabilities. The code and benchmark are publicly released.

Abstract

Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored. To address this gap, we introduce MECoBench, a multimodal embodied cooperation benchmark with an evaluation platform spanning diverse real-world tasks, two cooperation structures, and three collaboration modes. Through extensive experiments across various MLLMs, we summarize three key findings: (i) Collaboration generally improves embodied task completion, but its benefits depend on balancing collaborative gains against coordination complexity. (ii) Communication is essential to collaboration gains, while the best collaboration mode depends on team size and model capability. (iii) Moreover, collaboration improves robustness under noisy priors and exploration conditions. Generally, MECoBench provides a systematic testbed for understanding the mechanisms and limits of multimodal embodied collaboration. Code and dataset are available at https://github.com/q-i-n-g/MECoBench.

Problem

• multimodal agent collaboration
• embodied environments
• multimodal large language models
• cooperation benchmark
• visually grounded tasks

Innovation

• multimodal embodied collaboration
• MECoBench
• multimodal large language models
• cooperation benchmark
• embodied agents