Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation

2026-06-27

AI Summary

This work formalizes natural language prompt-based multi-agent routing as a set-valued prediction problem and introduces a weighted evaluation framework that jointly considers accuracy, latency, capability coverage, and execution cost. The authors construct a controlled multi-label benchmark comprising 3,000 prompts derived from WildChat, enhanced with AI-assisted annotation and rebalancing strategies. Experimental results demonstrate that supervised routing substantially outperforms nearest-neighbor and zero-shot large language model baselines. Among evaluated approaches, a fine-tuned encoder achieves the highest accuracy in unconstrained settings, while a linear multi-label model serves as the strongest practical baseline. Incorporating cost-aware Weighted Agent Routing (WAR) effectively improves utility under resource constraints, with the Encoder+WAR combination yielding the most significant performance gains.

Abstract

Tool and agent routing from natural-language prompts is naturally a set-valued prediction problem: a single query may require multiple agents, while over-selection increases execution cost. The benchmark introduced here is derived from WildChat and contains 3,000 prompts over a fixed 12-agent catalog, with AI-assisted heuristic labels under a fixed schema and controlled rebalancing for multi-label evaluation. The evaluation protocol combines set-level metrics (Precision, Recall, F1, Jaccard, and Exact Match), latency, an execution-oriented capability-coverage simulation, and a constrained weighted-routing setting based on ordinal agent-cost tiers. Compared methods include nearest-neighbor matching, linear multilabel classification, dependency-aware baselines, a fine-tuned encoder, deterministic weighted post-scoring via Weighted Agent Routing (WAR), and a zero-shot LLM baseline. Results show that supervised routers substantially outperform nearest-neighbor and zero-shot LLM routing. The fine-tuned encoder achieves the strongest unconstrained set accuracy, while the linear multilabel model provides the strongest practical baseline. In the constrained setting, the weighted routing layer improves utility when applied on top of strong supervised scorers, with the largest gain observed for Encoder+WAR. Overall, the benchmark and evaluation protocol support reproducible study of accuracy-cost trade-offs in fixed-catalog multi-agent routing.

Problem

• multi-agent routing
• set-valued prediction
• cost-aware evaluation
• agent selection
• natural-language prompts

Innovation

• set-valued prediction
• multi-agent routing
• cost-aware evaluation
• Weighted Agent Routing (WAR)
• WildChat benchmark