Fair Decisions from Calibrated Scores: Achieving Optimal Classification While Satisfying Sufficiency

2026-02-07

AI Summary

This work addresses the challenge of achieving optimal binary classification under the sufficiency fairness constraint—requiring equal positive predictive value (PPV) and false omission rate (FOR) across groups—while preserving prediction consistency and calibrated scores. The authors propose a post-processing randomized classification method that leverages group-calibrated scores and group identity to provide, for the first time, a precise geometric characterization and exact construction of an optimal randomized classifier satisfying sufficiency. By analyzing the geometry of the feasible PPV-FOR region, the method strictly enforces sufficiency without sacrificing classification performance. Moreover, when sufficiency conflicts with separation, it yields a Pareto-optimal trade-off with minimal deviation from separation. Experiments on real calibrated scores demonstrate that the algorithm effectively maintains predictive fairness while achieving near-optimal performance when balancing both fairness criteria.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Binary classification based on predicted probabilities (scores) is a fundamental task in supervised machine learning. While thresholding scores is Bayes-optimal in the unconstrained setting, using a single threshold generally violates statistical group fairness constraints. Under independence (statistical parity) and separation (equalized odds), such thresholding suffices when the scores already satisfy the corresponding criterion. However, this does not extend to sufficiency: even perfectly group-calibrated scores -- including true class probabilities -- violate predictive parity after thresholding. In this work, we present an exact solution for optimal binary (randomized) classification under sufficiency, assuming finite sets of group-calibrated scores. We provide a geometric characterization of the feasible pairs of positive predictive value (PPV) and false omission rate (FOR) achievable by such classifiers, and use it to derive a simple post-processing algorithm that attains the optimal classifier using only group-calibrated scores and group membership. Finally, since sufficiency and separation are generally incompatible, we identify the classifier that minimizes deviation from separation subject to sufficiency, and show that it can also be obtained by our algorithm, often achieving performance comparable to the optimum.

Problem

• fairness
• sufficiency
• calibrated scores
• binary classification
• predictive parity

Innovation

• sufficiency
• group calibration
• fair classification
• post-processing algorithm
• predictive parity