From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?

2026-04-02

AI Summary

This work addresses the challenges of high coordination overhead, fragmented context, and fragile sequential dependencies in multi-agent systems, as well as the lack of principled guidance in existing single-agent distillation approaches. The authors propose a two-stage adaptive distillation framework that introduces, for the first time, “Metric Freedom” (F) as a prior predictive indicator of skill utility. They demonstrate that F is governed by the topological rigidity of evaluation metrics and construct it by quantifying the coupling between output diversity and score variance via Mantel tests. Leveraging F, the framework dynamically selects between knowledge extraction and iterative optimization strategies. Experiments across four tasks, eleven datasets, and six metrics reveal a significant negative correlation between F and skill utility (ρ = −0.62, p < 0.05), with the proposed method achieving comparable or superior performance while reducing computational cost by up to 8× and latency by up to 15×.

Abstract

Multi-agent systems (MAS) tackle complex tasks by distributing expertise, though this often comes at the cost of heavy coordination overhead, context fragmentation, and brittle phase ordering. Distilling a MAS into a single-agent skill can bypass these costs, but this conversion lacks a principled answer for when and what to distill. Instead, the empirical outcome is surprisingly inconsistent: skill lift ranges from a 28% improvement to a 2% degradation across metrics of the exact same task. In this work, we reveal that skill utility is governed not by the task, but by the evaluation metric. We introduce Metric Freedom ($F$), the first a priori predictor of skill utility. $F$ measures the topological rigidity of a metric's scoring landscape by quantifying how output diversity couples with score variance via a Mantel test. Guided by $F$, we propose a two-stage adaptive distillation framework. Stage 1 acts as a selective extraction mechanism, extracting tools and knowledge while discarding restrictive structures on "free" metrics to preserve exploration. Stage 2 targets computationally intensive iterative refinement exclusively toward "rigid" metrics ($F \lesssim 0.6$) to eliminate trajectory-local overfitting. Evaluating across 4 tasks, 11 datasets, and 6 metrics, $F$ strongly predicts skill utility ($ρ= -0.62$, $p < 0.05$). Strikingly, identical agent trajectories yield diametrically opposite skill lifts under rigid versus free metrics, demonstrating that skill utility is fundamentally a metric-level property. Driven by this signal, our adaptive agent matches or exceeds the original MAS while reducing cost up to 8$\times$ and latency by up to 15$\times$.

Problem

• skill distillation
• multi-agent systems
• evaluation metric
• metric freedom
• single-agent

Innovation

• Metric Freedom
• Skill Distillation
• Multi-Agent Systems
• Adaptive Distillation
• Evaluation Metric Rigidity