CIFA: Contextual-Intersectional Fairness Auditing for Hidden Subgroup Discovery in Face Analysis

2026-08-10

AI Summary

Current evaluations of fairness in vision models predominantly rely on overall accuracy and analysis along a single demographic attribute, often failing to uncover hidden disadvantaged subgroups arising from interactions between contextual factors and demographic attributes. This work proposes CIFA, a framework that systematically integrates demographic and contextual dimensions to identify and rank the most vulnerable attribute combinations through intersectional fairness auditing, thereby revealing performance disparities obscured by conventional methods. Experiments on FairFace, CelebA, and UTKFace datasets using ResNet-50 and ViT-B/16 demonstrate that CIFA effectively uncovers significant intersectional unfairness. Although several mitigation strategies show partial efficacy, none consistently eliminate worst-group performance gaps across diverse datasets and model architectures.

Abstract

Fairness evaluation in computer vision commonly relies on aggregate accuracy and demographic subgroup analysis. However, visual models are also sensitive to contextual factors such as illumination, blur, image quality, facial accessories, and appearance attributes. These factors may interact with demographic characteristics, producing hidden subgroups in which performance degrades substantially despite strong aggregate accuracy and apparently acceptable demographic fairness. To address this, we propose the Contextual-Intersectional Fairness Auditing Framework (CIFA), a structured framework for identifying subgroup vulnerabilities arising from interactions between demographic and contextual attributes. CIFA performs demographic, contextual, and contextual-intersectional auditing, followed by worst-group discovery to identify and rank the most vulnerable attribute combinations. We evaluate CIFA on gender classification using ResNet-50 \cite{he2016deep} and ViT-B/16 \cite{dosovitskiy2020image} across FairFace \cite{Karkkainen2021}, CelebA \cite{Liu2015}, and UTKFace \cite{Zhang2017}. Our results show that aggregate accuracy and demographic-only evaluation can mask substantial contextual-intersectional disparities. We further assess several established mitigation strategies through an audit--mitigate--reaudit protocol and find that, although some worst-group disparities are reduced, no single strategy consistently eliminates them across datasets and architectures. These findings establish contextual-intersectional auditing as an important component of fairness evaluation and provide a reproducible framework for discovering, prioritizing, and reassessing hidden subgroup risks in face analysis systems.

Problem

• fairness auditing
• hidden subgroups
• contextual factors
• intersectional fairness
• face analysis

Innovation

• contextual-intersectional fairness
• hidden subgroup discovery
• fairness auditing
• face analysis
• worst-group identification