Switchcraft: AI Model Router for Agentic Tool Calling

2026-05-07

AI Summary

This work addresses the high inference costs incurred by agentic AI systems due to their overreliance on large language models for tool invocation. The authors propose the first online model routing mechanism specifically optimized for tool calling, which dynamically selects the lowest-cost model while preserving task accuracy. Built upon a lightweight DistilBERT-based classifier, the approach operates under latency constraints and is evaluated across five function-calling benchmarks. Experimental results demonstrate that the system achieves an accuracy of 82.9%—comparable to the best single-model performance—while reducing inference costs by 84%, translating to savings exceeding $3,600 per million queries. This study reveals that smaller models can outperform larger ones on specific tool-related tasks and pioneers the integration of total operational cost, rather than model price alone, into routing decisions.

Abstract

Agentic AI systems that invoke external tools are powerful but costly, leading developers to default to large models and overspend inference budgets. Model routing can mitigate this, but existing routers are designed for chat completion rather than tool use. We present Switchcraft, the first (to the best of our knowledge) model router optimized for agentic tool calling. Switchcraft operates inline, selecting the lowest-cost model subject to correctness. We construct an evaluation framework on five function-calling benchmarks and train a DistilBERT-based classifier, deployed under a latency budget. Switchcraft achieves 82.9% accuracy -- matching or exceeding the best individual model -- while reducing inference cost by 84%, saving over $3,600 per million queries. We find that larger models do not consistently outperform smaller ones on tool-use tasks, and that nominally cheaper models can incur higher total cost due to token-intensive reasoning. Our work enables cost-aware agentic AI deployment without sacrificing correctness.

Problem

• agentic AI
• tool calling
• model routing
• inference cost
• cost-efficiency

Innovation

• model routing
• agentic AI
• tool calling
• cost optimization
• DistilBERT classifier