🤖 AI Summary
This study addresses the common tendency in existing research to attribute minor performance differences among multi-agent large language model (LLM) coordination architectures to their inherent merits, while overlooking the impact of random noise. To rigorously isolate architectural effects, we introduce a paired experimental protocol that strictly controls model and task configurations, enabling the first quantification of a noise baseline in coordination mechanisms. We further propose a pass^k reporting standard that scores only when coordination logic is actively engaged. Leveraging SHA-256 auditing, Wilson confidence intervals, and coordination activation detection hooks, our experiments on Claude Haiku 4.5 reveal performance fluctuations due to noise ranging from –3 to +18 percentage points (with a combined upper bound of approximately 15 pp). Notably, most recently reported gains fall below this noise floor, and their coordination mechanisms show no significant improvement in task recovery rates on first attempts.
📝 Abstract
Multi-agent LLM coordination papers report small benchmark deltas as evidence that one architecture beats another. A prior question: how much paired trial-0 disagreement do two protocols produce on the same model and benchmark when their API inputs are configuration-equivalent (matched by code inspection plus a SHA-256 byte audit), short of full identity-replay? On Claude Haiku 4.5 against tau^2-bench retail, the clean configuration-equivalent contrast (no_coord vs. intercept, both inert at trial 0) gives signed paired gaps of +10pp and 0pp across two n=100 seeds; pooled across both, +5pp with Wilson CI [-2,+12], not significant. The largest single-seed contrast (+18pp pull-vs-intercept, p_corr=0.012) did not reproduce at the second seed (-3pp, p_corr=1.0); no trial-0 contrast is significant after Bonferroni at either seed or pooled. The envelope of observed paired gaps spans [-3,+18]pp across two seeds, with pooled upper Wilson CI ~15pp. Seven of ten recent multi-agent coordination architectures report headline effects below this local floor, and one more sits inside the envelope; whether they survive a same-model paired replication is, by construction, untested in their original settings.
We define coordination-active pass^k, pass^k restricted to trials where the coordination mechanism is logically active, as the minimum reporting protocol, with sample-size targets and runtime hooks in the body. Measurements run on ET-MCP, a task-scoped negative-knowledge store conformant with MCP 2026-07-28, used as a substrate to isolate reader-side choices, not as a contribution. On Haiku 4.5 the candidate readers (pull, intercept) do not improve trial-1 recovery; we give a preliminary diagnosis of failure modes with refinements on existing production hook surfaces.