Evaluating Fairness Metrics Across Borders from Human Perceptions

2024-03-24

AI Summary

The cross-cultural applicability of fairness metrics remains questionable, as single metrics often fail to generalize across diverse sociocultural contexts. Method: We conducted a stratified, cross-national survey involving 4,000 participants from China, France, Japan, and the United States. Using a structured scenario-based questionnaire, we assessed public preferences for four mainstream fairness metrics across three distinct decision-making domains. Statistical modeling and intersectional analysis examined the joint effects of nationality and individual attributes (e.g., age, education level). Contribution/Results: This study provides the first empirical evidence that national cultural background exerts systematic and statistically significant influence on fairness judgments—challenging the implicit “one-size-fits-all” assumption underlying many fairness metrics. It further delivers actionable, empirically grounded guidance for culturally adaptive AI governance and localized algorithmic design, proposing a framework for context-sensitive fairness operationalization.

Technology Category

• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Which fairness metrics are appropriately applicable in your contexts? There may be instances of discordance regarding the perception of fairness, even when the outcomes comply with established fairness metrics. Several questionnaire-based surveys have been conducted to evaluate fairness metrics with human perceptions of fairness. However, these surveys were limited in scope, including only a few hundred participants within a single country. In this study, we conduct an international survey to evaluate public perceptions of various fairness metrics in decision-making scenarios. We collected responses from 1,000 participants in each of China, France, Japan, and the United States, amassing a total of 4,000 participants, to analyze the preferences of fairness metrics. Our survey consists of three distinct scenarios paired with four fairness metrics. This investigation explores the relationship between personal attributes and the choice of fairness metrics, uncovering a significant influence of national context on these preferences.

Problem

• Assessing global applicability of fairness metrics in decision-making
• Investigating discrepancies between fairness metrics and human perceptions
• Analyzing national context influence on fairness metric preferences

Innovation

• Conducted international survey with 4,000 participants
• Compared fairness metrics across four countries
• Analyzed personal attributes and national context