Who's the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration

2024-12-13

Proceedings of the AAAI Conference on Artificial Intelligence

AI Summary

Existing interpolation-based fair data augmentation methods (e.g., Fair Mixup) are evaluated without accounting for model uncertainty and predominantly validated on datasets with relatively large minority subgroups; meanwhile, mainstream multicalibration-based post-processing requires withholding training data, rendering it inapplicable in settings with sparse minority groups. Method: This work introduces multicalibration as a rigorous evaluation framework—first applied to interpolation-based augmentation—and systematically assesses classification fairness across structured datasets comprising up to 81 marginalized subgroups. Contribution/Results: We find that Fair Mixup consistently degrades both multicalibration compliance and overall accuracy, whereas vanilla Mixup yields superior performance. Building on this insight, we propose a novel paradigm: “vanilla Mixup + multicalibration post-processing.” This approach preserves overall accuracy while reducing multicalibration violations among minority groups by up to 37% and improving average accuracy by 2.3%.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Data augmentation methods, especially SoTA interpolation-based methods such as Fair Mixup, have been widely shown to increase model fairness. However, this fairness is evaluated on metrics that do not capture model uncertainty and on datasets with only one, relatively large, minority group. As a remedy, multicalibration has been introduced to measure fairness while accommodating uncertainty and accounting for multiple minority groups. However, existing methods of improving multicalibration involve reducing initial training data to create a holdout set for post-processing, which is not ideal when minority training data is already sparse. This paper uses multicalibration to more rigorously examine data augmentation for classification fairness. We stress-test four versions of Fair Mixup on two structured data classification problems with up to 81 marginalized groups, evaluating multicalibration violations and balanced accuracy. We find that on nearly every experiment, Fair Mixup worsens baseline performance and fairness, but the simple vanilla Mixup outperforms both Fair Mixup and the baseline, especially when calibrating on small groups. Combining vanilla Mixup with multicalibration post-processing, which enforces multicalibration through post-processing on a holdout set, further increases fairness.

Problem

• Evaluates fairness in data augmentation using multicalibration metrics
• Tests Fair Mixup on datasets with multiple minority groups
• Compares vanilla Mixup and Fair Mixup for fairness performance

Innovation

• Uses multicalibration to evaluate fairness rigorously
• Tests Fair Mixup and vanilla Mixup on structured data
• Combines vanilla Mixup with multicalibration post-processing