Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models

2024-11-14

arXiv.org

AI Summary

To address the challenge of jointly optimizing quality and inference cost in foundation model (FM)-driven software, this paper proposes a real-time adaptive routing mechanism—the first to co-evolve routing decisions with weak-FM capability enhancement. Methodologically, it leverages cross-domain general guidance generated by strong FMs, enables parameter-free capability leap for weak FMs via Guided In-Context Learning (G-ICL), and integrates hierarchical FM collaborative inference with online request scheduling optimization. Evaluated on an MMLU subset, the approach reduces costly large-model invocations by 50.2% while maintaining 90.5% overall response quality; the guidance strategy exhibits strong cross-subdomain generalization. The core contribution lies in transcending traditional static routing paradigms by enabling joint dynamic evolution of routing policies and FM capabilities—thereby establishing a new framework for adaptive, cost-aware FM orchestration.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

To balance the quality and inference cost of a Foundation Model (FM, such as large language models (LLMs)) powered software, people often opt to train a routing model that routes requests to FMs with different sizes and capabilities. Existing routing models rely on learning the optimal routing decision from carefully curated data, require complex computations to be updated, and do not consider the potential evolution of weaker FMs. In this paper, we propose Real-time Adaptive Routing (RAR), an approach to continuously adapt FM routing decisions while using guided in-context learning to enhance the capabilities of weaker FM. The goal is to reduce reliance on stronger, more expensive FMs. We evaluate our approach on different subsets of the popular MMLU benchmark. Over time, our approach routes 50.2% fewer requests to computationally expensive models while maintaining around 90.5% of the general response quality. In addition, the guides generated from stronger models have shown intra-domain generalization and led to a better quality of responses compared to an equivalent approach with a standalone weaker FM.

Problem

• Balancing quality and cost in Foundation Model routing
• Adapting routing decisions in real-time for efficiency
• Enhancing weaker models via guided in-context learning

Innovation

• Real-time Adaptive Routing for FM efficiency
• Guided in-context learning enhances weaker FMs
• Reduces reliance on expensive FMs significantly