LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders

2025-09-28

AI Summary

This work addresses semantic bias introduced by pretrained text encoders in text-to-image diffusion models (T2I DMs). We propose an efficient, lightweight fairness-enhancement method. Methodologically, it features: (1) a collaborative distance-constrained debiasing strategy that explicitly models and suppresses bias directions associated with sensitive attributes in the CLIP embedding space; and (2) a two-stage text-guided sampling scheme—requiring no auxiliary networks—that jointly optimizes bias mitigation and generation fidelity. Crucially, our approach only fine-tunes the text encoder, avoiding full model retraining or complex auxiliary modules. Evaluated on Stable Diffusion v1.5, it achieves state-of-the-art fairness performance—improving FID and reducing Bias Score—while reducing training overhead to one-quarter of prior methods and preserving sampling efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper explores a novel lightweight approach LightFair to achieve fair text-to-image diffusion models (T2I DMs) by addressing the adverse effects of the text encoder. Most existing methods either couple different parts of the diffusion model for full-parameter training or rely on auxiliary networks for correction. They incur heavy training or sampling burden and unsatisfactory performance. Since T2I DMs consist of multiple components, with the text encoder being the most fine-tunable and front-end module, this paper focuses on mitigating bias by fine-tuning text embeddings. To validate feasibility, we observe that the text encoder's neutral embedding output shows substantial skewness across image embeddings of various attributes in the CLIP space. More importantly, the noise prediction network further amplifies this imbalance. To finetune the text embedding, we propose a collaborative distance-constrained debiasing strategy that balances embedding distances to improve fairness without auxiliary references. However, mitigating bias can compromise the original generation quality. To address this, we introduce a two-stage text-guided sampling strategy to limit when the debiased text encoder intervenes. Extensive experiments demonstrate that LightFair is effective and efficient. Notably, on Stable Diffusion v1.5, our method achieves SOTA debiasing at just $1/4$ of the training burden, with virtually no increase in sampling burden. The code is available at https://github.com/boyuh/LightFair.

Problem

• Addressing text encoder bias in diffusion models
• Reducing training burden for fair image generation
• Maintaining generation quality during debiasing process

Innovation

• Debiasing pre-trained text encoders for fairness
• Collaborative distance-constrained embedding optimization strategy
• Two-stage text-guided sampling for quality preservation