SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Tasks

2026-08-12

AI Summary

This work addresses critical limitations in existing vision-language model (VLM) routing methods, which are constrained by conventional VQA-based evaluation, lack open-set calibration, and ignore inference costs—leading to the dilution of multi-positive signals under softmax normalization. To overcome these issues, the authors propose SCOPE-Router, introducing the first execution-oriented VLM routing benchmark, VLM-ExecRouterBench; a training-free dual-tower routing architecture coupled with a hybrid calibration mechanism based on behavioral profiling; and a cost-aware objective function (CRM+RCCR) that preserves multi-positive signals while embedding cost preferences. Experiments demonstrate that SCOPE-Router achieves state-of-the-art Rank Scores across all three execution-domain benchmarks, outperforming the next-best method by 1.84 points under out-of-distribution (OOD) settings and by 6.75 points in dual-OOD evaluation, while consistently improving Rank Scores by 1.25–6.21 points across four diverse routing models.

Abstract

Model routing aims to select the most suitable model from a candidate pool for each query, balancing quality and cost. Existing VLM routing research is limited to traditional VQA evaluation, lacks systematic calibration optimization for open-set scenarios, and employs training objectives that dilute multi-positive signals via softmax normalization without incorporating cost. We address these limitations with three contributions: (1)VLM-ExecRouterBench, the first execution-oriented VLM routing benchmark covering Code, Agentic, and Search domains with 11 candidate models spanning nearly two orders of magnitude in pricing; (2)SCOPE-Router, a dual-tower router that matches queries to model behavior profiles constructed via hybrid calibration (random/diagnostic/diversity sampling), enabling new models to join routing without retraining; (3)CRM+RCCR, an architecture-agnostic cost-aware objective that encodes cost preference into continuous relevance targets through per-pair independent scoring, eliminating multi-positive dilution while regularizing queries with similar routing preferences to be closer in the routing space. Empirically, SCOPE-Router achieves the best Rank Score on all three benchmarks, surpassing the runner-up by 1.84 points under OOD settings and by 6.75 points under doubly OOD open-set evaluation. When applied to four diverse routers, CRM+RCCR improves Rank Score by 1.25--6.21 points.

Problem

• VLM routing
• open-set scenarios
• cost-aware
• multi-positive signals
• execution-oriented tasks

Innovation

• cost-aware routing
• open-set VLM
• model behavior profiling
• CRM+RCCR objective
• execution-oriented benchmark