All Eyes on the Ranker: Participatory Auditing to Surface Blind Spots in Ranked Search Results

2026-04-10

AI Summary

This study addresses the limitations of traditional search evaluation, which often overlooks users’ lived experiences concerning accountability, harm, and trust in ranked results, thereby failing to uncover critical societal blind spots. The authors propose the first user-perceived impact taxonomy encompassing cognitive, representational, infrastructural, and downstream social dimensions. They design a participatory auditing workshop featuring a customizable search interface with adjustable transparency, integrating BM25 and MonoT5 to enable users to compare ranking models, manipulate results, and reflect on causal impacts. The research identifies accountability gaps invisible to conventional audits and reveals that high-trust neural rankers can suppress users’ critical scrutiny, leading them to overlook manipulative behaviors. These findings underscore both the value and limitations of participatory approaches in evaluating algorithmic systems.

Technology Category

• Search and Optimization: Algorithm Configuration
• Machine Learning: Learning Preferences or Rankings
• Humans and AI: User Experience and Usability

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Algorithmic accountability and transparency on the web
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Search engines that present users with a ranked list of search results are a fundamental technology for providing public access to information. Evaluations of such systems are typically conducted by domain experts and focus on model-centric metrics, relevance judgments, or output-based analyses, rather than on how accountability, harm, or trust are experienced by users. This paper argues that participatory auditing is essential for revealing users'causal and contextual understandings of how ranked search results produce impacts, particularly as ranking models appear increasingly convincing and sophisticated in their semantic interpretation of user queries. We report on three participatory auditing workshops (n=21) in which participants engaged with a custom search interface across four tasks, comparing a lexical ranker (BM25) and a neural semantic reranker (MonoT5), exploring varying levels of transparency and user controls, and examining an intentionally adversarially manipulated ranking. Reflexive activities prompted participants to articulate causal narratives linking search system properties to broader impacts. Synthesising the findings, we contribute a taxonomy of user-perceived impacts of ranked search results, spanning epistemic, representational, infrastructural, and downstream social impacts. However, interactions with the neural model revealed limits to participatory auditing itself: perceived system competence and accumulated trust reduced critical scrutiny during the workshop, allowing manipulations to go undetected. Participants expressed desire for visibility into the full search pipeline and recourse mechanisms. Together, these findings show how participatory auditing can surface user perceived impacts and accountability gaps that remain unseen when relying on conventional audits, while revealing where participatory auditing may encounter limitations.

Problem

• participatory auditing
• ranked search results
• user-perceived impacts
• accountability gaps
• search engine bias

Innovation

• participatory auditing
• ranked search results
• user-perceived impacts
• neural semantic reranker
• algorithmic accountability