Search results diversification in competitive search

2025-01-24

AI Summary

This paper addresses the problem of result homogenization and diversity degradation in competitive web search, caused by content authors strategically imitating top-ranked pages. To mitigate this, we propose a game-aware ranking mechanism that jointly optimizes relevance and diversity. We establish, for the first time, the theoretical existence of a Nash equilibrium under diversity-aware ranking in competitive settings, and formulate a game-theoretic joint optimization model balancing diversity and relevance. By designing a novel evaluation function and an efficient equilibrium-finding algorithm, we empirically demonstrate that our mechanism significantly curbs content imitation (reduction of 32.7%), improves retrieval coverage (+18.4%), and enhances ranking fairness (+24.1%). Our core contribution lies in proving the equilibrium existence of diversity-oriented ranking under competition and validating its tangible positive impact on content ecosystem health.

Technology Category

• Game Theory and Economic Paradigms: Mechanism Design
• Multiagent Systems: Mechanism Design
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

In Web retrieval, there are many cases of competition between authors of Web documents: their incentive is to have their documents highly ranked for queries of interest. As such, the Web is a prominent example of a competitive search setting. Past work on competitive search focused on ranking functions based solely on relevance estimation. We study ranking functions that integrate a results-diversification aspect. We show that the competitive search setting with diversity-based ranking has an equilibrium. Furthermore, we theoretically and empirically show that the phenomenon of authors mimicking content in documents highly ranked in the past, which was demonstrated in previous work, is mitigated when search results diversification is applied.

Problem

• Diversity Enhancement
• Redundancy Reduction
• Content Imitation Avoidance

Innovation

• Diversified Ranking
• Competitive Search Equilibrium
• Reduced Content Imitation