Welfarist Formulations for Diverse Similarity Search

2026-02-09

AI Summary

This work addresses the challenge of dynamically balancing relevance and diversity in similarity search, a task where traditional methods often fall short. It introduces, for the first time, social welfare functions—satisfying fairness and efficiency axioms—into nearest neighbor search, proposing a tunable, parameterized framework that optimizes Nash social welfare to achieve query-dependent trade-offs adaptively. The approach is designed to be seamlessly integrated as a subroutine into any approximate nearest neighbor (ANN) algorithm and includes an efficient approximation strategy for practical deployment. Experimental results demonstrate that the method significantly enhances result diversity while maintaining high relevance, offering both strong theoretical guarantees and practical utility.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Learning Preferences or Rankings
• Constraint Satisfaction and Optimization: Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Social mining and social search on the Web

Abstract

Nearest Neighbor Search (NNS) is a fundamental problem in data structures with wide-ranging applications, such as web search, recommendation systems, and, more recently, retrieval-augmented generations (RAG). In such recent applications, in addition to the relevance (similarity) of the returned neighbors, diversity among the neighbors is a central requirement. In this paper, we develop principled welfare-based formulations in NNS for realizing diversity across attributes. Our formulations are based on welfare functions -- from mathematical economics -- that satisfy central diversity (fairness) and relevance (economic efficiency) axioms. With a particular focus on Nash social welfare, we note that our welfare-based formulations provide objective functions that adaptively balance relevance and diversity in a query-dependent manner. Notably, such a balance was not present in the prior constraint-based approach, which forced a fixed level of diversity and optimized for relevance. In addition, our formulation provides a parametric way to control the trade-off between relevance and diversity, providing practitioners with flexibility to tailor search results to task-specific requirements. We develop efficient nearest neighbor algorithms with provable guarantees for the welfare-based objectives. Notably, our algorithm can be applied on top of any standard ANN method (i.e., use standard ANN method as a subroutine) to efficiently find neighbors that approximately maximize our welfare-based objectives. Experimental results demonstrate that our approach is practical and substantially improves diversity while maintaining high relevance of the retrieved neighbors.

Problem

• Nearest Neighbor Search
• Diversity
• Relevance
• Retrieval-Augmented Generation
• Welfare

Innovation

• welfare-based formulation
• Nash social welfare
• diverse similarity search
• adaptive relevance-diversity trade-off
• approximate nearest neighbor