RouteLLM: Learning to Route LLMs with Preference Data

2024-06-26

arXiv.org

AI Summary

This work addresses the challenge of balancing cost and response quality in multi-LLM serving systems. We propose a lightweight, human-preference-driven dynamic routing method that trains a generalizable LLM router—e.g., a LoRA-finetuned TinyBERT—to automatically select strong or weak base models per input, achieving substantial cost reduction without compromising quality. Our key contributions are: (1) the first zero-shot cross-model routing framework leveraging human preference data, eliminating reliance on fixed model deployments; and (2) integration of synthetic data augmentation to enhance router generalization. Experiments across multiple benchmarks demonstrate over 2× cost savings while matching the response quality of the strongest single LLM. Moreover, the router maintains over 92% accuracy when deployed with unseen underlying LLMs, confirming strong transferability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost. More powerful models, though effective, come with higher expenses, while less capable models are more cost-effective. To address this dilemma, we propose several efficient router models that dynamically select between a stronger and a weaker LLM during inference, aiming to optimize the balance between cost and response quality. We develop a training framework for these routers leveraging human preference data and data augmentation techniques to enhance performance. Our evaluation on widely-recognized benchmarks shows that our approach significantly reduces costs-by over 2 times in certain cases-without compromising the quality of responses. Interestingly, our router models also demonstrate significant transfer learning capabilities, maintaining their performance even when the strong and weak models are changed at test time. This highlights the potential of these routers to provide a cost-effective yet high-performance solution for deploying LLMs.

Problem

• Optimize cost-quality trade-off in LLM usage
• Dynamic selection between strong and weak LLMs
• Leverage human preference data for training routers

Innovation

• Dynamic LLM selection router
• Training with human preference data
• Cost reduction without quality loss