The Magnitude Mirage: Rethinking Confidence for Reasoning-Intensive Retrieval

2026-09-14

AI Summary

研究解决了基于相似度得分阈值的检索回避在需要推理的任务中失效的问题，通过使用零成本的查询性能预测指标如Score Gap和LSMV来改进。

Abstract

Many production RAG systems implement retrieval abstention by thresholding raw similarity scores, implicitly treating score magnitude as a confidence signal. We demonstrate that this practice degrades systematically as queries require reasoning beyond semantic matching. Across 11 retrieval architectures and 28 datasets, neural retrievers consistently assign high similarity scores to semantically related but constraint-violating documents, causing magnitude-based thresholds to collapse toward near-random abstention performance on logical and temporal reasoning tasks---a failure we term the Magnitude Mirage. To address this without computationally expensive alternatives, we conduct a large-scale empirical study of six zero-cost Query Performance Prediction (QPP) metrics across three cognitive tiers: semantic matching (BEIR), logical reasoning (BRIGHT), and temporal reasoning (TEMPO). Our central finding is that the key improvement comes from abandoning magnitude in favor of score-distribution signals: the gain from this shift exceeds the differences among distributional alternatives by a factor of 5-10$\times$. In particular, Score Gap ($s_1 - s_k$) and a practical adaptation of Score Magnitude and Variance (LSMV) improve abstention AUROC by up to 0.16 in settings where magnitude-based confidence provides little discriminative power. These methods require no additional inference, retraining, or latency, making them a practical zero-cost replacement for magnitude thresholding in deployed RAG systems.

Problem

• retrieval abstention
• similarity scores
• reasoning tasks
• magnitude-based thresholds
• semantic matching

Innovation

• Score Gap
• Score Magnitude and Variance
• abstention AUROC
• Magnitude Mirage
• Query Performance Prediction