Think When Needed: Model-Aware Reasoning Routing for LLM-based Ranking

2026-01-26

AI Summary

This work addresses the high computational cost and unstable gains of chain-of-thought (CoT) reasoning in large language models (LLMs) for ranking tasks. To mitigate these issues, the authors propose a lightweight reasoning routing framework that dynamically selects between direct reasoning and CoT before generation, based on candidate dispersion and model-perceived difficulty signals. The framework incorporates a plug-in router head, ranking-aware feature extraction, and a controllable token mechanism to allocate computational resources on demand. It further enables adaptive strategy adjustment along the validation Pareto frontier during deployment. Experiments on three public ranking benchmarks demonstrate significant performance improvements with reduced overhead; for instance, on MovieLens, Qwen3-4B achieves a 6.3% gain in NDCG@10 while reducing token consumption by 49.5%.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) are increasingly applied to ranking tasks in retrieval and recommendation. Although reasoning prompting can enhance ranking utility, our preliminary exploration reveals that its benefits are inconsistent and come at a substantial computational cost, suggesting that when to reason is as crucial as how to reason. To address this issue, we propose a reasoning routing framework that employs a lightweight, plug-and-play router head to decide whether to use direct inference (Non-Think) or reasoning (Think) for each instance before generation. The router head relies solely on pre-generation signals: i) compact ranking-aware features (e.g., candidate dispersion) and ii) model-aware difficulty signals derived from a diagnostic checklist reflecting the model's estimated need for reasoning. By leveraging these features before generation, the router outputs a controllable token that determines whether to apply the Think mode. Furthermore, the router can adaptively select its operating policy along the validation Pareto frontier during deployment, enabling dynamic allocation of computational resources toward instances most likely to benefit from Think under varying system constraints. Experiments on three public ranking datasets with different scales of open-source LLMs show consistent improvements in ranking utility with reduced token consumption (e.g., +6.3\% NDCG@10 with -49.5\% tokens on MovieLens with Qwen3-4B), demonstrating reasoning routing as a practical solution to the accuracy-efficiency trade-off.

Problem

• reasoning routing
• LLM-based ranking
• computational efficiency
• accuracy-efficiency trade-off
• model-aware reasoning

Innovation

• reasoning routing
• model-aware difficulty
• plug-and-play router
• accuracy-efficiency trade-off
• LLM-based ranking