🤖 AI Summary
Machine learning models exhibit high sensitivity to minor perturbations in training data, leading to unstable predictions; yet conventional fairness metrics (e.g., bias-based indicators) ignore this prediction uncertainty. Method: We propose a variance-oriented paradigm for group fairness—introducing the first systematic framework that treats inter-group predictive variance equality as a core fairness criterion, grounded in statistical error decomposition and theoretical analysis of variance’s independent impact on fairness assessment. Contribution/Results: We release VarFair, the first open-source library integrating uncertainty quantification with fairness evaluation. Extensive experiments on Adult, COMPAS, and other benchmarks demonstrate that groups with high predictive variance are frequently misclassified as “fair” by standard methods, whereas our variance-aware metric significantly improves identification of disadvantaged groups and enhances assessment robustness under data perturbations.
📝 Abstract
The error of an estimator can be decomposed into a (statistical) bias term, a variance term, and an irreducible noise term. When we do bias analysis, formally we are asking the question:"how good are the predictions?"The role of bias in the error decomposition is clear: if we trust the labels/targets, then we would want the estimator to have as low bias as possible, in order to minimize error. Fair machine learning is concerned with the question:"Are the predictions equally good for different demographic/social groups?"This has naturally led to a variety of fairness metrics that compare some measure of statistical bias on subsets corresponding to socially privileged and socially disadvantaged groups. In this paper we propose a new family of performance measures based on group-wise parity in variance. We demonstrate when group-wise statistical bias analysis gives an incomplete picture, and what group-wise variance analysis can tell us in settings that differ in the magnitude of statistical bias. We develop and release an open-source library that reconciles uncertainty quantification techniques with fairness analysis, and use it to conduct an extensive empirical analysis of our variance-based fairness measures on standard benchmarks.
