RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing

2026-06-17

AI Summary

This work addresses the lack of reproducible, user-preference-driven evaluation frameworks in existing large language model (LLM) routing systems. We propose the first evaluation paradigm specifically focused on the routing layer, introducing RouteJudge—an online pairwise preference assessment platform that anonymously collects user comparisons of responses generated under different routing strategies. By attributing user preferences directly to routing decisions and integrating multidimensional factors such as cost, latency, and task metadata, our approach enables nuanced analysis. Complementing this, we develop ORBIT, a modular open-source toolbox offering unified interfaces for query representation, routing algorithm implementation, budget-aware evaluation, and method comparison, thereby supporting end-to-end standardized research workflows. Together, RouteJudge and ORBIT establish an open, extensible, and preference-aware ecosystem for LLM routing evaluation.

Abstract

We present RouteJudge, an online pairwise preference evaluation framework for LLM routing systems, with a public platform available at https://routejudge.cn. Different from model-level response evaluation, RouteJudge focuses on router-level decision quality. For each user query, multiple routing strategies independently recommend candidate models under the same model pool and budget constraints. The selected model responses are then presented to users through anonymous pairwise comparisons, and the resulting user preferences are attributed back to the routing strategies behind the compared responses. Each evaluation record stores the query, routing decisions, model responses, preference labels, cost, latency, and task metadata, enabling preference-aware, cost-aware, and task-conditioned analysis of LLM routers. To support the continuous expansion of routing methods in RouteJudge, we further release ORBIT (Optimal Routing and Budgeted Inference Toolbox), a modular and extensible toolbox that standardizes the end-to-end workflow of LLM routing. ORBIT provides unified interfaces for benchmark loading, query representation, router implementation, budget-aware evaluation, and method comparison, allowing researchers to develop and evaluate routing algorithms under consistent protocols. It also serves as the submission and integration layer for RouteJudge: researchers can implement routing methods within ORBIT, validate them on existing routing benchmarks, and submit compatible routers for online preference-based evaluation. The code of ORBIT is available at https://github.com/AIGNLAI/LAMDA-ORBIT.

Problem

• LLM routing
• preference evaluation
• reproducibility
• router-level decision
• cost-aware evaluation

Innovation

• LLM routing
• preference-based evaluation
• RouteJudge
• ORBIT
• budget-aware inference