Fair Generation without Unfair Distortions: Debiasing Text-to-Image Generation with Entanglement-Free Attention

2025-06-16

AI Summary

Text-to-image (T2I) diffusion models often generate images that reinforce societal stereotypes due to embedded biases, and existing debiasing methods suffer from attribute entanglement—altering non-target attributes (e.g., background, pose) when intervening on target attributes (e.g., race), causing distributional shift. To address this, we propose Entanglement-Free Attention (EFA), the first method enabling fully decoupled control: it precisely modulates target attributes while preserving non-target ones throughout the diffusion process. EFA achieves this via cross-attention intervention, stochastic sampling of target attributes, and layer-wise selective reweighting. Crucially, fairness improvement is attained without compromising fidelity. Experiments across multi-dimensional bias benchmarks demonstrate that EFA significantly outperforms state-of-the-art methods: target-attribute fairness improves by up to 42%, while non-target attribute fidelity remains at 98.3% of the original model’s performance.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advancements in diffusion-based text-to-image (T2I) models have enabled the generation of high-quality and photorealistic images from text descriptions. However, they often exhibit societal biases related to gender, race, and socioeconomic status, thereby reinforcing harmful stereotypes and shaping public perception in unintended ways. While existing bias mitigation methods demonstrate effectiveness, they often encounter attribute entanglement, where adjustments to attributes relevant to the bias (i.e., target attributes) unintentionally alter attributes unassociated with the bias (i.e., non-target attributes), causing undesirable distribution shifts. To address this challenge, we introduce Entanglement-Free Attention (EFA), a method that accurately incorporates target attributes (e.g., White, Black, Asian, and Indian) while preserving non-target attributes (e.g., background details) during bias mitigation. At inference time, EFA randomly samples a target attribute with equal probability and adjusts the cross-attention in selected layers to incorporate the sampled attribute, achieving a fair distribution of target attributes. Extensive experiments demonstrate that EFA outperforms existing methods in mitigating bias while preserving non-target attributes, thereby maintaining the output distribution and generation capability of the original model.

Problem

• Mitigate societal biases in text-to-image generation
• Prevent attribute entanglement during bias correction
• Preserve non-target attributes while adjusting target ones

Innovation

• Introduces Entanglement-Free Attention (EFA) for debiasing
• Preserves non-target attributes during bias mitigation
• Adjusts cross-attention layers for fair attribute distribution