Stop Overthinking: Unlocking Efficient Listwise Reranking with Minimal Reasoning

2026-05-14

AI Summary

This work addresses the high computational cost of existing reasoning-augmented list re-ranking models, which suffer from “overthinking” due to generating excessively long reasoning texts. The authors propose a length-regularized self-distillation framework that, for the first time, reveals a diminishing marginal return between reasoning length and ranking performance in re-ranking tasks. By introducing a Pareto-optimal heuristic to identify efficient reasoning trajectories, the framework guides the student model to learn concise yet effective reasoning patterns. Evaluated on the TREC Deep Learning and NeuCLIR benchmarks, the method reduces reasoning token consumption by 34%–37% while preserving the teacher model’s ranking effectiveness, thereby establishing a lightweight inference paradigm that balances efficacy and efficiency.

Abstract

Listwise reranking utilizing Large Language Models (LLMs) has achieved state-of-the-art retrieval effectiveness. Recently, reasoning-enhanced models have further pushed these boundaries by employing Chain-of-Thought (CoT) to perform deep comparative analysis of candidate documents. However, this performance gain comes at a prohibitive computational cost, as models often generate thousands of reasoning tokens before producing a final ranking. In this work, we investigate the relationship between reasoning length and ranking quality, revealing an overthinking phenomenon where extended reasoning yields diminishing returns. To address this, we propose a Length-Regularized Self-Distillation framework. We synthesize a dataset by sampling diverse reasoning traces from a teacher model (Rank-K) and applying a Pareto-inspired filter to select traces that achieve high ranking performance with minimal token usage. By fine-tuning on these concise, high-quality rationales, the student model learns to internalize efficient reasoning patterns, effectively pruning redundant deliberation. Experiments on TREC Deep Learning and NeuCLIR benchmarks demonstrate that our method maintains the teacher's effectiveness while reducing inference token consumption by 34%-37% across different retrieval settings, offering a practical solution for deploying reasoning-enhanced rerankers in latency-sensitive applications.

Problem

• listwise reranking
• reasoning efficiency
• overthinking
• Large Language Models
• token consumption

Innovation

• Length-Regularized Self-Distillation
• Listwise Reranking
• Reasoning Efficiency
• Overthinking Phenomenon
• Pareto-inspired Filtering