IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory

2025-06-01

AI Summary

This work addresses the query-to-model dynamic routing problem in multi-LLM settings, proposing an interpretable matching method that jointly optimizes performance and cost. Methodologically, it introduces Item Response Theory (IRT)—a psychometric framework—into LLM routing for the first time, modeling the intrinsic relationship between model capability and query difficulty to enable quantifiable, interpretable capability/difficulty assessment. It further designs a semantic-embedding-based online warm-start mechanism to mitigate cold-start issues. The approach comprises four components: IRT-based modeling, maximum-likelihood capability estimation, semantic similarity computation, and adaptive warm-up scheduling. Extensive experiments across 20 LLMs and 12 benchmark datasets demonstrate significant improvements over state-of-the-art baselines: up to 23.6% higher accuracy under cold-start conditions, while maintaining high efficiency, robustness, and strong interpretability.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have demonstrated exceptional performance across a wide range of natural language tasks. However, selecting the optimal LLM to respond to a user query often necessitates a delicate balance between performance and cost. While powerful models deliver better results, they come at a high cost, whereas smaller models are more cost-effective but less capable. To address this trade-off, we propose IRT-Router, a multi-LLM routing framework that efficiently routes user queries to the most suitable LLM. Inspired by Item Response Theory (IRT), a psychological measurement methodology, IRT-Router explicitly models the relationship between LLM capabilities and user query attributes. This not only enables accurate prediction of response performance but also provides interpretable insights, such as LLM abilities and query difficulty. Additionally, we design an online query warm-up technique based on semantic similarity, further enhancing the online generalization capability of IRT-Router. Extensive experiments on 20 LLMs and 12 datasets demonstrate that IRT-Router outperforms most baseline methods in terms of effectiveness and interpretability. Its superior performance in cold-start scenarios further confirms the reliability and practicality of IRT-Router in real-world applications. Code is available at https://github.com/Mercidaiha/IRT-Router.

Problem

• Balancing performance and cost in LLM selection
• Modeling LLM capabilities and query attributes
• Enhancing routing accuracy and interpretability

Innovation

• Uses Item Response Theory for LLM routing
• Models LLM capabilities and query attributes
• Online query warm-up enhances generalization