RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

2026-08-16

AI Summary

This study addresses the suboptimal decision-making and performance-cost trade-off challenges in large model cascade routing caused by quality prediction errors. We propose a quality-estimator-free framework that formulates routing as a Markov Decision Process, employing reinforcement learning to directly optimize the target objective. By jointly modeling stopping and selection actions through a policy network, this approach eliminates reliance on independent posterior estimation. Experiments across ten benchmarks demonstrate that the proposed framework significantly outperforms strong baselines, achieving superior performance-cost trade-offs while supporting zero-shot integration of unseen models. Consequently, this work establishes a novel paradigm for efficient and flexible large model routing.

Abstract

The growing ecosystem of large language models (LLMs) offers huge potential to optimize performance-cost trade-offs. However, their heterogeneous capabilities and inference costs make efficiently routing queries a significant challenge. Existing paradigms are inflexible: one-shot routers commit before observing responses, whereas conventional cascades stop adaptively but follow a fixed model order. Cascade routing removes both restrictions by reconsidering whether to stop or invoke another model after each response. Current methods use a predict-then-optimize pipeline estimating response quality and future model utility. However, prediction loss for quality or utility is not equivalent to routing-decision loss. A lower prediction error does not necessarily yield a better action; a small boundary-crossing error can reverse a ``stop'' or model-selection decision. Therefore, we propose RLCascadeRouter, a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising ``stop'' and model selection. It uses trajectory returns and advantages to directly optimize the performance-cost objective. Its Cascade Policy Network models candidate complementarity for model selection and remaining-action value for stopping, eliminating independent post-hoc response-quality estimators. Evaluated across ten LLMRouterBench benchmarks with thirteen LLMs, RLCascadeRouter outperforms strong baselines and achieves superior performance-cost trade-offs. It incorporates unseen models without retraining, and ablation studies validate both policy components.

Problem

• Cascade Routing
• Large Language Models
• Performance-Cost Trade-off
• Predict-then-Optimize
• Routing Decision Loss

Innovation

• Quality-Estimator-Free
• Reinforcement Learning
• Cascade Routing
• Markov Decision Process
• Performance-Cost Trade-off