Problem
Research questions and friction points this paper is trying to address.
Cost Reduction
Large Language Models
Cloud API
AutoMix: Automatically Mixing Language Models
🤖 AI Summary
To address the challenge of balancing computational cost and performance in collaborative deployment of multi-scale large language models (LLMs), this paper proposes a lightweight, training-free dynamic routing framework. Methodologically, it introduces (1) a few-shot self-verification mechanism that assesses output reliability via consistency scoring, and (2) a confidence-aware routing policy grounded in partially observable Markov decision processes (POMDPs), enabling small models to perform preliminary inference, self-verify outputs, and trigger large-model invocation only when necessary—forming a cascaded decision pipeline. Experiments across five mainstream LLMs and five challenging benchmarks demonstrate that our approach maintains or even improves task performance while reducing computational cost by over 50%, significantly outperforming existing model selection and cascading baselines.
📝 Abstract
Large language models (LLMs) are now available from cloud API providers in various sizes and configurations. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present Automix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to Automix are two key technical contributions. First, it has a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring extensive training. Second, given that self-verification can be noisy, it employs a POMDP based router that can effectively select an appropriately sized model, based on answer confidence. Experiments across five language models and five challenging datasets show that Automix consistently surpasses strong baselines, reducing computational cost by over 50% for comparable performance.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Automix
POMDP-based routing
reliability assessment
🔎 Similar Papers
No similar papers found.
