De-centering the (Traditional) User: Multistakeholder Evaluation of Recommender Systems

2025-01-09

AI Summary

Existing evaluation paradigms for multi-stakeholder recommender systems suffer from misalignment due to conflicting values and heterogeneous objectives among users, producers, platforms, and other stakeholders. Method: We propose the first structured multi-stakeholder evaluation framework, integrating value-sensitive design, multi-objective trade-off modeling, and domain-adapted metric engineering to establish an end-to-end pathway—from value alignment to actionable metrics—and define cross-scenario generalizable evaluation dimensions. Contribution: Our work breaks the dominant single-stakeholder utility paradigm by establishing a theory–metric–empirical closed loop; we release an extensible evaluation guideline covering three representative domains—education, e-commerce, and content platforms; and we set a new community benchmark for multi-stakeholder evaluation in RecSys.

Technology Category

• Multiagent Systems: Mechanism Design
• Data Mining & Knowledge Management: Recommender Systems
• Game Theory and Economic Paradigms: Mechanism Design

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems

Abstract

Multistakeholder recommender systems are those that account for the impacts and preferences of multiple groups of individuals, not just the end users receiving recommendations. Due to their complexity, evaluating these systems cannot be restricted to the overall utility of a single stakeholder, as is often the case of more mainstream recommender system applications. In this article, we focus our discussion on the intricacies of the evaluation of multistakeholder recommender systems. We bring attention to the different aspects involved in the evaluation of multistakeholder recommender systems - from the range of stakeholders involved (including but not limited to producers and consumers) to the values and specific goals of each relevant stakeholder. Additionally, we discuss how to move from theoretical principles to practical implementation, providing specific use case examples. Finally, we outline open research directions for the RecSys community to explore. We aim to provide guidance to researchers and practitioners about how to think about these complex and domain-dependent issues of evaluation in the course of designing, developing, and researching applications with multistakeholder aspects.

Problem

• Multistakeholder Evaluation
• Diverse Needs
• Recommender Systems

Innovation

• Diverse Stakeholder Needs
• Multi-Criteria Evaluation
• Recommendation Systems