Explaining When PRF Fails: Participatory Auditing for Selective Query Expansion

2026-07-13

AI Summary

This study addresses the inconsistent impact of pseudo-relevance feedback (PRF) in information retrieval, which, despite improving average performance, often degrades user experience due to query drift and lacks interpretability in existing selective PRF approaches. The authors propose an “audit–automation” two-stage framework: first, a user-centered audit involving 108 participants quantifies PRF’s real-world effects across diverse queries; second, a large language model (LLM)-based reranker automatically predicts user preferences while providing verifiable rationales for its decisions. This work pioneers participatory auditing in IR evaluation, revealing that only 20.9% of queries benefit from PRF, 25.6% suffer degradation, and avoiding harm is roughly twice as valuable as achieving gains. The proposed method faithfully replicates human judgments at scale, enabling interpretable and auditable selective PRF.

Abstract

Pseudo-Relevance Feedback (PRF) improves retrieval effectiveness on average, but harms a substantial fraction of queries through query drift, an asymmetry hidden by aggregate offline metrics. Existing Selective PRF (sPRF) approaches typically rely on Query Performance Prediction (QPP) methods derived from the same ranking statistics, and therefore inherit, rather than resolve, this opacity. We argue that this is a core explainability problem in IR, and propose a two-stage audit-then-automate framework. In Stage 1, a participatory audit with 108 users across 43 TREC Deep Learning 2019 queries shows that only 20.9% of queries benefit from PRF, while 25.6% suffer a degraded user experience, and that avoiding harm is nearly twice as valuable as exploiting successful expansion. In Stage 2, we repurpose LLM-based rerankers as system preference predictors that replicate these user-derived labels automatically, grounded in inspectable document evidence. Together, the two stages explain which queries PRF harms, why an sPRF decision is made, and how the decision can be inspected at scale, turning an opaque retrieval component into an auditable, user-grounded one.

Problem

• Pseudo-Relevance Feedback
• Query Drift
• Explainability
• Selective Query Expansion
• Information Retrieval

Innovation

• participatory auditing
• selective pseudo-relevance feedback
• query drift
• LLM-based reranking
• explainable IR