A Theoretical Framework for Risk Analysis of Stochastic Rankers

2026-06-15

AI Summary

This work addresses the issue that stochastic re-ranking strategies may induce worst-case fluctuations in retrieval effectiveness prior to re-ranking. It presents the first theoretical framework to quantify the maximum absolute change in Discounted Cumulative Gain (DCG)—termed “re-ranking risk”—caused by such randomness. This risk is determined by the positional distribution of relevant documents in the initial retrieval list. By integrating probability theory with ranking metric analysis, the study performs an extremal analysis of DCG variation and derives a computable upper bound for this risk. Experiments on the TREC Fairness 2022 dataset demonstrate strong alignment between theoretical predictions and observed DCG fluctuations, thereby filling a critical theoretical gap in pre-re-ranking risk assessment.

Abstract

Different from deterministic rankers that seek to maximize relevance at top ranks, stochastic ranking policies instead estimate distributions over permutations, from which rankings are sampled, towards obtaining diversified or fair exposure. Such policies are commonly evaluated in terms of expected effectiveness postreranking. However, the randomness inherent in these policies gives rise to a fundamental but under-explored ex ante question: prior to applying stochastic reranking, how large can the induced variation in retrieval effectiveness be in the worst case? This paper presents a theoretical analysis of reranking risk, defined as the maximum absolute change in discounted cumulative gain (DCG) resulting from a permutation sampled from a stochastic reranking policy applied to a fixed retrieved list.We derive that this risk is governed by the distribution of the recall points in the initial retrieved list. We conduct experiments on submitted runs from the TREC Fairness 2022 track that employ stochastic reranking policies and empirically demonstrate that the effectiveness variations predicted by our theory closely approximate the observed changes in DCG.

Problem

• stochastic rankers
• reranking risk
• discounted cumulative gain
• effectiveness variation
• risk analysis

Innovation

• stochastic rankers
• reranking risk
• discounted cumulative gain
• theoretical analysis
• fairness in ranking