When is Routing Meaningful? Diversity and Robustness in Language Model Societies

2026-07-10

AI Summary

This work addresses a critical limitation in existing routing strategies for language model societies, which predominantly prioritize task accuracy and inference cost while neglecting behavioral diversity and routing stability—often resulting in meaningless routing decisions. To remedy this, the study introduces Hierarchical Social Entropy (HSE) into language model societies and proposes a perturbation-based robustness metric to evaluate routing effectiveness along two dimensions: behavioral diversity and routing robustness. Through comprehensive analyses involving HSE, query perturbations, and comparisons between KNN and prompt-based routing, the authors demonstrate that fewer than ten carefully selected agents suffice to capture the primary diversity of a model pool. While KNN improves accuracy, it exhibits poor robustness; in contrast, prompt-based routing remains stable under various perturbations, revealing a fundamental tension between accuracy and the meaningfulness of routing outcomes.

Abstract

Routing policies for multi-model systems are evaluated almost exclusively on task accuracy and inference cost. We argue that two properties, orthogonal to performance, determine whether routing is meaningful. First, the society of actors must be behaviourally differentiated: if all actors respond identically, routing is vacuous. Second, the routing policy must be stable: surface-form variants of a query should be assigned to the same actor. High task accuracy is compatible with violating both properties, since a router can operate over a redundant society or assign queries inconsistently, preventing specialisation regardless of performance. We adapt Hierarchic Social Entropy (HSE) to language-model societies and introduce a perturbation-based robustness metric to diagnose these failure modes. Applied to EmbedLLM and RouterBench, we find that HSE exhibits strong diminishing returns, suggesting that a curated subset of fewer than ten agents recovers most available diversity in a large pool -- a practical coreset heuristic for society design. We further find that KNN routers gain accuracy from specialist societies but collapse in robustness under perturbation, while prompted routing remains stable across all perturbation types -- illustrating that accuracy and meaningfulness can sharply diverge.

Problem

• routing meaningfulness
• behavioral diversity
• routing robustness
• language model societies
• specialization

Innovation

• routing meaningfulness
• behavioral diversity
• routing robustness
• Hierarchic Social Entropy
• language model societies