Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space

2026-01-09

arXiv.org

AI Summary

This work addresses the pervasive model lock-in problem in large language model (LLM) ecosystems, where integrating new models incurs substantial retraining costs. To overcome this, the authors propose ZeroRouter, a novel framework that introduces a model-agnostic universal latent space. By employing a context-aware predictor to map queries into this shared space, ZeroRouter decouples query characteristics from individual model performance, enabling zero-shot integration of new models without retraining. The framework further incorporates a dual-mode optimizer that dynamically balances accuracy, inference cost, and latency according to deployment requirements. Extensive experiments demonstrate that ZeroRouter significantly outperforms existing routing methods across multiple benchmarks, achieving higher routing accuracy at reduced computational cost and lower latency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

The rapid proliferation of Large Language Models (LLMs) has led to a fragmented and inefficient ecosystem, a state of ``model lock-in''where seamlessly integrating novel models remains a significant bottleneck. Current routing frameworks require exhaustive, costly retraining, hindering scalability and adaptability. We introduce ZeroRouter, a new paradigm for LLM routing that breaks this lock-in. Our approach is founded on a universal latent space, a model-agnostic representation of query difficulty that fundamentally decouples the characterization of a query from the profiling of a model. This allows for zero-shot onboarding of new models without full-scale retraining. ZeroRouter features a context-aware predictor that maps queries to this universal space and a dual-mode optimizer that balances accuracy, cost, and latency. Our framework consistently outperforms all baselines, delivering higher accuracy at lower cost and latency.

Problem

• model lock-in
• LLM routing
• zero-shot onboarding
• scalability
• retraining cost

Innovation

• universal latent space
• zero-shot routing
• model-agnostic representation
• LLM routing
• cost-efficient inference