Do Counterfactually Fair Image Classifiers Satisfy Group Fairness? -- A Theoretical and Empirical Study

2026-07-06

AI Summary

This work addresses the unclear relationship between counterfactual fairness (CF) and group fairness (GF) in image classification, as well as the lack of real-world counterfactual samples for evaluating CF. To this end, we construct two high-quality image datasets, OursCeleb and OursLFW, leveraging high-fidelity image editing and human annotation. Using these datasets, we reveal— for the first time in vision tasks—that CF does not imply GF, and propose a latent variable G to explain this inconsistency. We further introduce Counterfactual Knowledge Distillation (CKD), a method designed to reduce model reliance on sensitive-attribute-related latent variables. Experiments demonstrate that applying CKD on our datasets enables models satisfying CF to also achieve GF, significantly improving the alignment between these two notions of fairness.

Abstract

The notion of algorithmic fairness has been actively explored from various aspects of fairness, such as counterfactual fairness (CF) and group fairness (GF). However, the exact relationship between CF and GF remains to be unclear, especially in image classification tasks; the reason is because we often cannot collect counterfactual samples regarding a sensitive attribute, essential for evaluating CF, from the existing images (\eg, a photo of the same person but with different secondary sex characteristics). In this paper, we construct new image datasets for evaluating CF by using a high-quality image editing method and carefully labeling with human annotators. Our datasets, \oursceleb and \ourslfw, build upon the popular image GF benchmarks; hence, we can evaluate CF and GF simultaneously. We empirically observe that CF does not imply GF in image classification, whereas previous studies on tabular datasets observed the opposite. We theoretically show that it could be due to the existence of a latent attribute $G$ that is correlated with, but not caused by, the sensitive attribute (\eg, secondary sex characteristics are highly correlated with hair length). From this observation, we propose a simple baseline, Counterfactual Knowledge Distillation (CKD), to mitigate such correlation with the sensitive attributes. Extensive experimental results on \oursceleb and \ourslfw demonstrate that CF-achieving models satisfy GF if we successfully reduce the reliance on $G$ (\eg, using CKD).

Problem

• counterfactual fairness
• group fairness
• image classification
• sensitive attribute
• algorithmic fairness

Innovation

• counterfactual fairness
• group fairness
• image dataset construction
• latent attribute
• knowledge distillation