BAFIS: Dataset + Framework to assess occupational Bias and Human Preference in modern Text-to-image Models
This study addresses the significant gender and racial biases exhibited by contemporary text-to-image models in generating occupation-related imagery, which conventional evaluation metrics often fail to capture due to their inability to reflect human subjective judgments of fairness. To bridge this gap, the authors propose BAFIS—a fairness evaluation framework that integrates human preference feedback with multilingual prompts—and construct a dataset of 21,140 images aligned with official employment statistics. Using this framework, they systematically assess occupational bias, image quality, and prompt alignment across leading models including Midjourney v6.1, Stable Diffusion 3 Medium, and DALL·E 3. The work pioneers the incorporation of human preference annotations into bias evaluation, revealing systematic disparities in model outputs and demonstrating only partial correlation between human feedback and traditional automated metrics, thereby underscoring the critical role of human judgment in developing equitable text-to-image generation systems.