AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

2025-06-17

AI Summary

Distilling LLM-based agents—incorporating planning, memory, and tool invocation—into small language models (SLMs) remains highly inefficient due to reliance on costly supervised fine-tuning or reinforcement learning. Method: This paper introduces the first training-free agent knowledge distillation framework. It formalizes teacher LLM-generated, structured task knowledge as modular, domain-agnostic Model-Context-Protocol (MCP) units (“MCP Boxes”), enabling zero-gradient, trajectory-free knowledge transfer via protocol-driven encapsulation and plug-and-play reuse. Contribution/Results: By eliminating parameter updates and trajectory replay, the method drastically reduces deployment cost and inference overhead. Evaluated on biomedical and mathematical benchmarks, distilled SLM-based agents achieve performance on par with the GPT-4o-powered OctoTools system, demonstrating strong effectiveness, scalability, and practical deployability.

Technology Category

• Multiagent Systems: Modeling other Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

While knowledge distillation has become a mature field for compressing large language models (LLMs) into smaller ones by aligning their outputs or internal representations, the distillation of LLM-based agents, which involve planning, memory, and tool use, remains relatively underexplored. Existing agent distillation methods typically replay full teacher trajectories or imitate step-by-step teacher tool usage, but they often struggle to train student agents to dynamically plan and act in novel environments. We propose AgentDistill, a novel, training-free agent distillation framework that enables efficient and scalable knowledge transfer via direct reuse of Model-Context-Protocols (MCPs), which are structured and reusable task-solving modules autonomously generated by teacher agents. The reuse of these distilled MCPs enables student agents to generalize their capabilities across domains and solve new problems with minimal supervision or human intervention. Experiments on biomedical and mathematical benchmarks demonstrate that our distilled student agents, built on small language models, can achieve performance comparable to advanced systems using large LLMs such as OctoTools (GPT-4o), highlighting the effectiveness of our framework in building scalable and cost-efficient intelligent agents.

Problem

• Distilling LLM-based agents with planning and tool use capabilities
• Enabling student agents to generalize across domains with minimal supervision
• Achieving performance comparable to large LLMs using small language models

Innovation

• Training-free agent distillation framework
• Reuse of Model-Context-Protocols (MCPs)
• Generalizable task-solving across domains