Fairness Beyond Demographics: Optimizing Performance Across Appearance-Based Hidden Cohorts in Medical Imaging

2026-05-28

AI Summary

This work addresses performance disparities of medical imaging models across patient subgroups, a challenge inadequately tackled by conventional fairness methods that rely on explicit demographic labels and struggle with data sparsity in multi-attribute settings while overlooking image-appearance-defined latent subgroups. To overcome these limitations, the authors propose Label-agnostic Hidden-queue Fairness (LHCF), a novel training paradigm that leverages unsupervised image clustering to construct K appearance-driven hidden queues and enforces fairness constraints over them—eliminating the need for any demographic annotations while optimizing fairness across both single and multiple attributes. Building upon this approach, they introduce HIDFairBench, a benchmark for evaluating fairness in medical imaging. Experiments demonstrate that LHCF substantially reduces performance gaps across subgroups, achieving state-of-the-art fairness with strong scalability and robustness.

Abstract

Medical image analysis models can exhibit performance disparities across patient subgroups, threatening clinical safety and fairness. Existing methods typically address this issue by optimizing accuracy and fairness metrics for visible demographic attributes (e.g., sex or age) considered in isolation. This strategy not only overlooks potentially more informative latent stratifications, which may reveal deeper sources of model error and inequity, but also fails to scale when multiple demographic attributes are considered simultaneously due to the resulting sparsity of training data within each subgroup. We deal with these issues by introducing the label-free hidden-cohort fairness (LHCF) training paradigm that instead of maximizing fairness over visible demographic attributes, it optimizes fairness across latent subpopulations discovered from image appearance. By clustering images into K appearance-based cohorts and applying fairness optimization over them, LHCF uncovers underlying sources of model error and avoids the combinatorial sparsity of multi-demographic attributes, reducing disparities across both single and multiple demographic attributes. We demonstrate on our proposed fairness benchmark, HIDFairBench, that LHCF provides state-of-the-art fairness results on single and multiple demographic attributes, despite never using demographic labels for training. Our results position hidden-cohort fairness as a practical, scalable, and robust alternative to demographic-based fairness optimization for trustworthy medical image analysis.

Problem

• medical imaging
• fairness
• hidden cohorts
• demographic attributes
• performance disparity

Innovation

• hidden-cohort fairness
• medical image analysis
• label-free fairness
• appearance-based clustering
• demographic-agnostic optimization