T-Retrievability: A Topic-Focused Approach to Measure Fair Document Exposure in Information Retrieval

2025-08-29

AI Summary

Existing global-retrievability-based measures of document exposure fairness conflate exposure bias with topic-specific relevance differences, leading to distorted evaluations. To address this, we propose T-Retrievability—a topic-aware, local retrievability metric: documents are first clustered by topic; within each cluster, local retrievability is computed; finally, these local scores are aggregated into a global fairness indicator. Unlike conventional global approaches, T-Retrievability enables precise detection of latent exposure disparities across topics induced by neural ranking models. Experiments demonstrate that T-Retrievability significantly enhances the granularity and robustness of fairness analysis, effectively uncovering subtle yet critical exposure differences among models. It thus establishes a more reliable, interpretable, and topic-sensitive benchmark for fairness assessment in information retrieval.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Topic discovery and tracking

Abstract

Retrievability of a document is a collection-based statistic that measures its expected (reciprocal) rank of being retrieved within a specific rank cut-off. A collection with uniformly distributed retrievability scores across documents is an indicator of fair document exposure. While retrievability scores have been used to quantify the fairness of exposure for a collection, in our work, we use the distribution of retrievability scores to measure the exposure bias of retrieval models. We hypothesise that an uneven distribution of retrievability scores across the entire collection may not accurately reflect exposure bias but rather indicate variations in topical relevance. As a solution, we propose a topic-focused localised retrievability measure, which we call extit{T-Retrievability} (topic-retrievability), which first computes retrievability scores over multiple groups of topically-related documents, and then aggregates these localised values to obtain the collection-level statistics. Our analysis using this proposed T-Retrievability measure uncovers new insights into the exposure characteristics of various neural ranking models. The findings suggest that this localised measure provides a more nuanced understanding of exposure fairness, offering a more reliable approach for assessing document accessibility in IR systems.

Problem

• Measuring exposure bias in retrieval models using retrievability distribution
• Proposing topic-focused retrievability to assess document exposure fairness
• Evaluating neural ranking models with localized topic-based retrievability measure

Innovation

• Topic-focused localized retrievability measure
• Computes scores over topically-related document groups
• Aggregates localized values for collection statistics