ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents

2026-06-30

AI Summary

Existing benchmarks inadequately assess large language models’ (LLMs’) ability to coordinate subagents and construct dynamic workflows as managers. This work proposes the first benchmark specifically designed to evaluate LLM team-management capabilities, comprising 41 multi-turn, multimodal, cross-directory scenarios in which a constrained main agent orchestrates a fixed pool of local subagents to complete tasks. The study introduces an innovative Subagent-Management Score (SMS)—an LLM-free evaluation metric that combines principle-of-least-privilege access control with modality-based routing to enable execution-driven automatic assessment. Experiments reveal that the primary bottleneck lies in permission granting rather than perceptual capabilities; management quality can be decoupled from computational cost, with certain open-source models performing competitively; and despite similar overall scores, orchestration behaviors across models differ by over an order of magnitude.

Abstract

Production large language-model (LLM) agents are increasingly deployed not as lone problem-solvers but as managers: a main model creates specialized subagents, delegates work, and orchestrates their parallel, asynchronous returns through dynamic workflows. Whether one model can actually run such a team is largely unmeasured: existing benchmarks score a policy's own task-solving or a fixed multi-agent system's emergent behavior, but none isolate the management ability of the single LLM acting as leader. We introduce ClawArena-Team, a benchmark of 41 multi-turn, multimodal, multi-directory scenarios spanning 258 evaluation rounds and 72 staged updates that measures this management ability. The main agent is deliberately constrained: it natively perceives only text and directly accesses only part of the workspace. It commands a fixed, locally served subagent pool, so score differences reflect management skill, not raw capability. All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor. Across twelve proprietary, community-hosted, and self-hosted models, experiments show that the management bottleneck is privilege granting rather than perception (no model exceeds 50% workspace-permission precision); that cost and management quality are decoupled (API cost spans over 100 times while the overall score spans under 4 times, with the cheapest open models on the Pareto frontier); and that most leaderboard scores cluster within a 9.9-point band while orchestration behaviors diverge by more than an order of magnitude. Code and data will be released.

Problem

• subagent orchestration
• dynamic workflows
• language-model agents
• management ability
• benchmarking

Innovation

• subagent orchestration
• dynamic workflows
• LLM agent benchmarking
• least-privilege routing
• execution-based evaluation