VL-RouterBench: A Benchmark for Vision-Language Model Routing

2025-12-29

AI Summary

Existing work lacks a systematic, reproducible benchmark for evaluating vision-language model (VLM) routing. Method: We introduce the first log-driven VLM routing benchmark, covering 14 datasets, 17 VLMs, and 30,540 samples, and construct a dual matrix quantifying per-sample quality and inference cost across models. We propose a multi-objective evaluation protocol integrating accuracy, cost, and throughput, normalized via harmonic mean ranking; further, we quantify— for the first time—the gap between router performance and the oracle upper bound (up to 23.6%), exposing bottlenecks in visual fine-grained understanding and textual structural modeling. Contribution/Results: We open-source a complete toolchain—including data generation, scoring, and visualization—and evaluate 10 routing methods on 519,180 sample-model pairs, demonstrating significant performance gains and identifying concrete avenues for improvement.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multi-model routing has evolved from an engineering technique into essential infrastructure, yet existing work lacks a systematic, reproducible benchmark for evaluating vision-language models (VLMs). We present VL-RouterBench to assess the overall capability of VLM routing systems systematically. The benchmark is grounded in raw inference and scoring logs from VLMs and constructs quality and cost matrices over sample-model pairs. In scale, VL-RouterBench covers 14 datasets across 3 task groups, totaling 30,540 samples, and includes 15 open-source models and 2 API models, yielding 519,180 sample-model pairs and a total input-output token volume of 34,494,977. The evaluation protocol jointly measures average accuracy, average cost, and throughput, and builds a ranking score from the harmonic mean of normalized cost and accuracy to enable comparison across router configurations and cost budgets. On this benchmark, we evaluate 10 routing methods and baselines and observe a significant routability gain, while the best current routers still show a clear gap to the ideal Oracle, indicating considerable room for improvement in router architecture through finer visual cues and modeling of textual structure. We will open-source the complete data construction and evaluation toolchain to promote comparability, reproducibility, and practical deployment in multimodal routing research.

Problem

• Lacks systematic benchmark for vision-language model routing evaluation
• Measures VLM routing systems' accuracy, cost, and throughput performance
• Identifies gap between current routers and ideal performance for improvement

Innovation

• Benchmark for vision-language model routing evaluation
• Constructs quality and cost matrices from sample-model pairs
• Uses harmonic mean of normalized cost and accuracy ranking