Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference

2026-09-07

AI Summary

为提高LLM推理效率，提出Signed Rescue Routing方法，通过预测大小模型纠错和误替事件差值来优化请求路由，优于传统基于不确定性的方法。

Abstract

Large language model (LLM) cascades answer easy requests with a small model and escalate selected requests to a larger model. Most routers prioritize examples on which the small model appears uncertain or likely to be wrong. This proxy ignores a decisive fact: escalation is useful only when the large model corrects the small model, and it is harmful when the large model replaces a correct answer with an incorrect one. We introduce Signed Rescue Routing (SRR), a budgeted routing method that predicts these two events separately and ranks requests by their difference. We show that this signed conditional gain is the Bayes-optimal routing score under a fixed escalation budget. SRR requires only the small model's output statistics at deployment and adds a lightweight two-head router. We evaluate SRR with Qwen3-4B and Qwen3-8B on TBD examples from MMLU, HellaSwag, and ARC-Challenge. Across the accuracy-compute curve, SRR reaches an area of TBD, compared with TBD for a learned small-model error predictor and TBD for entropy routing. These results show that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.

Problem

• Large Language Model
• Cascades
• Model Uncertainty
• Routing
• Efficiency

Innovation

• Signed Rescue Routing
• Bayes-optimal routing score
• incremental value prediction
• efficient LLM cascades