FairPlay: A Collaborative Approach to Mitigate Bias in Datasets for Improved AI Fairness

2025-04-22

AI Summary

AI fairness faces a fundamental challenge: stakeholders hold mutually incompatible conceptions of fairness, and no universally accepted theoretical foundation exists to reconcile them. Method: We propose a collaborative data debiasing framework grounded in a novel game-theoretic negotiation paradigm, enabling multi-stakeholder consensus without presupposing a unified fairness definition. Our approach implements a web-based application supporting real-time interaction, dynamically computing and visualizing multi-faceted fairness metrics, and simulating downstream impacts of debiasing interventions. Contribution/Results: A user study demonstrates that participants reach cross-group-acceptable debiasing solutions in an average of five negotiation rounds. This work transcends reliance on any single fairness theory, establishing a new paradigm for AI fairness governance—interpretable, iterative, and consensus-driven—while advancing practical, human-in-the-loop fairness engineering.

Technology Category

• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Game Theory and Economic Paradigms: Fair Division
• Humans and AI: Learning Human Values and Preferences

Application Category

• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

The issue of fairness in decision-making is a critical one, especially given the variety of stakeholder demands for differing and mutually incompatible versions of fairness. Adopting a strategic interaction of perspectives provides an alternative to enforcing a singular standard of fairness. We present a web-based software application, FairPlay, that enables multiple stakeholders to debias datasets collaboratively. With FairPlay, users can negotiate and arrive at a mutually acceptable outcome without a universally agreed-upon theory of fairness. In the absence of such a tool, reaching a consensus would be highly challenging due to the lack of a systematic negotiation process and the inability to modify and observe changes. We have conducted user studies that demonstrate the success of FairPlay, as users could reach a consensus within about five rounds of gameplay, illustrating the application's potential for enhancing fairness in AI systems.

Problem

• Mitigating bias in datasets for improved AI fairness
• Enabling collaborative stakeholder negotiation on fairness standards
• Providing a tool to reach consensus without universal fairness theory

Innovation

• Web-based tool for collaborative dataset debiasing
• Enables negotiation without universal fairness standards
• Facilitates consensus through interactive gameplay rounds