Revisiting the Berkeley Admissions data: Statistical Tests for Causal Hypotheses

2025-02-14

AI Summary

Correlation-based reasoning in fairness analysis is vulnerable to Simpson’s paradox, as exemplified by the 1973 Berkeley graduate admissions data. Method: This work reconstructs fairness definitions from a causal inference perspective, proposing a set of observable and testable causal fairness criteria. It rigorously establishes logical equivalences among graphical models, counterfactuals, and interventional fairness notions. Furthermore, it pioneers the adaptation of Pearl’s instrumental variable inequality into a statistical test for observational data. Contribution/Results: Empirical validation on the Berkeley dataset confirms consistency and testability across multiple causal fairness measures. The framework provides a rigorous, nonparametric, assumption-light causal foundation for algorithmic fairness assessment—requiring no strong modeling assumptions or untestable ignorability conditions.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Reasoning about fairness through correlation-based notions is rife with pitfalls. The 1973 University of California, Berkeley graduate school admissions case from Bickel et. al. (1975) is a classic example of one such pitfall, namely Simpson's paradox. The discrepancy in admission rates among males and female applicants, in the aggregate data over all departments, vanishes when admission rates per department are examined. We reason about the Berkeley graduate school admissions case through a causal lens. In the process, we introduce a statistical test for causal hypothesis testing based on Pearl's instrumental-variable inequalities (Pearl 1995). We compare different causal notions of fairness that are based on graphical, counterfactual and interventional queries on the causal model, and develop statistical tests for these notions that use only observational data. We study the logical relations between notions, and show that while notions may not be equivalent, their corresponding statistical tests coincide for the case at hand. We believe that a thorough case-based causal analysis helps develop a more principled understanding of both causal hypothesis testing and fairness.

Problem

• Addresses fairness pitfalls in correlation-based reasoning.
• Introduces causal hypothesis testing using Pearl's inequalities.
• Compares and tests causal fairness notions with observational data.

Innovation

• Causal hypothesis testing introduced
• Pearl's instrumental-variable inequalities utilized
• Observational data for fairness analysis