Measuring machine learning harms from stereotypes: requires understanding who is being harmed by which errors in what ways

2024-02-06

arXiv.org

AI Summary

Existing fairness metrics largely overlook subjective psychological harm, particularly lacking empirical investigation into stereotype-driven differential harms. Method: This study focuses on gender stereotypes in image search, integrating online surveys, randomized controlled experiments, qualitative interviews, and causal inference to systematically incorporate social-psychological stereotype theory into ML fairness evaluation for the first time. Contribution/Results: (1) Stereotype-reinforcing errors significantly intensify women’s subjective discomfort without altering their beliefs or behaviors; (2) certain stereotype-violating errors elicit stronger negative affect among men; (3) subjective harm dissociates from cognitive impact. These findings motivate a new “contextualized, subject-sensitive” fairness paradigm—shifting fair ML evaluation from statistical parity toward psychological justice.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

As machine learning applications proliferate, we need an understanding of their potential for harm. However, current fairness metrics are rarely grounded in human psychological experiences of harm. Drawing on the social psychology of stereotypes, we use a case study of gender stereotypes in image search to examine how people react to machine learning errors. First, we use survey studies to show that not all machine learning errors reflect stereotypes nor are equally harmful. Then, in experimental studies we randomly expose participants to stereotype-reinforcing, -violating, and -neutral machine learning errors. We find stereotype-reinforcing errors induce more experientially (i.e., subjectively) harmful experiences, while having minimal changes to cognitive beliefs, attitudes, or behaviors. This experiential harm impacts women more than men. However, certain stereotype-violating errors are more experientially harmful for men, potentially due to perceived threats to masculinity. We conclude that harm cannot be the sole guide in fairness mitigation, and propose a nuanced perspective depending on who is experiencing what harm and why.

Problem

• Understanding harm from ML errors based on stereotypes
• Assessing differential harm impacts on genders from errors
• Evaluating experiential harm versus cognitive belief changes

Innovation

• Survey studies differentiate harmful stereotype errors
• Experimental exposure to stereotype-reinforcing errors
• Nuanced harm perspective guides fairness mitigation