Universal Model Routing for Efficient LLM Inference

2025-02-12

AI Summary

This work addresses the problem of dynamically routing inference requests at test time to *unseen large language models (LLMs)* to reduce computational cost. Methodologically, it introduces the first general-purpose model routing framework: it extracts representative LLM feature vectors from a small set of prompts, constructs a generalizable model representation space via clustering, and incorporates a learnable cluster mapping mechanism coupled with statistical risk modeling—enabling zero-shot routing to over 30 LLMs excluded from training. Theoretically, it provides the first rigorous upper bound on routing error and proves optimality under mild assumptions. Empirically, the framework significantly reduces inference overhead across multiple public benchmarks while achieving routing accuracy close to the theoretical optimum, demonstrating strong generalization and reliability.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models' significant advances in capabilities are accompanied by significant increases in inference costs. Model routing is a simple technique for reducing inference cost, wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective strategies, relying on cluster-based routing and a learned cluster map respectively. We prove that these strategies are estimates of a theoretically optimal routing rule, and provide an excess risk bound to quantify their errors. Experiments on a range of public benchmarks show the effectiveness of the proposed strategies in routing amongst more than 30 unseen LLMs.

Problem

• Dynamic routing for unseen LLMs
• Efficient inference cost reduction
• Cluster-based routing strategies

Innovation

• Dynamic routing for LLMs
• Feature vector representation
• Cluster-based routing strategies