Diversification as Risk Minimization

2025-10-26

AI Summary

Existing diversified ranking methods in information retrieval struggle to cover niche user intents, while mainstream evaluation metrics focus solely on average relevance and lack mechanisms to control the risk of failure for worst-case intents. Method: We formulate diversified ranking as a risk minimization problem and propose VRisk—a novel metric that quantifies per-query robustness from the perspective of expected risk over the most vulnerable intent group. We theoretically prove that conventional algorithms offer no robustness advantage under this objective. Furthermore, we design VRisker, a greedy re-ranking algorithm with provable approximation guarantees and computational efficiency. Results: Extensive evaluation on NTCIR, TREC, and MovieLens datasets shows that VRisker reduces the failure rate for the worst-case intents by up to 33%, with only a marginal average performance degradation of 2%. This significantly enhances system robustness for multi-intent queries and strengthens the lower-bound guarantee on user experience.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Distributed Search
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Users tend to remember failures of a search session more than its many successes. This observation has led to work on search robustness, where systems are penalized if they perform very poorly on some queries. However, this principle of robustness has been overlooked within a single query. An ambiguous or underspecified query (e.g., ``jaguar'') can have several user intents, where popular intents often dominate the ranking, leaving users with minority intents unsatisfied. Although the diversification literature has long recognized this issue, existing metrics only model the average relevance across intents and provide no robustness guarantees. More surprisingly, we show theoretically and empirically that many well-known diversification algorithms are no more robust than a naive, non-diversified algorithm. To address this critical gap, we propose to frame diversification as a risk-minimization problem. We introduce VRisk, which measures the expected risk faced by the least-served fraction of intents in a query. Optimizing VRisk produces a robust ranking, reducing the likelihood of poor user experiences. We then propose VRisker, a fast greedy re-ranker with provable approximation guarantees. Finally, experiments on NTCIR INTENT-2, TREC Web 2012, and MovieLens show the vulnerability of existing methods. VRisker reduces worst-case intent failures by up to 33% with a minimal 2% drop in average performance.

Problem

• Addresses poor robustness in search diversification algorithms
• Proposes risk-minimization framework for ambiguous query intents
• Reduces worst-case failures for minority intents in rankings

Innovation

• Framing diversification as risk minimization problem
• Introducing VRisk metric for least-served intents
• Developing VRisker greedy reranker with approximation guarantees