Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization

2026-07-01

AI Summary

This work addresses the pervasive issue of gender and racial biases in text-to-image diffusion models, which often generate stereotypical outputs even from neutral prompts. To mitigate such biases without retraining the model, the authors propose a Text Embedding Steering (TES) framework that dynamically refines conditional text embeddings through a two-stage strategy. Initially, global alignment ensures semantic consistency with the input prompt; subsequently, iterative refinement guided by CLIP-based feedback fine-tunes the denoising process for precise control over sensitive attributes. Evaluated on Stable Diffusion, TES significantly outperforms existing training-free debiasing methods, achieving markedly improved fairness in generated images while preserving high visual quality and semantic fidelity to the original prompt.

Abstract

Text-to-image diffusion models achieve impressive visual quality, yet demographic bias remains a challenge, as neutral prompts consistently produce stereotypical representations across gender and race. Existing approaches remain limited by costly retraining or by inference-time interventions that often degrade image quality and semantic alignment. We propose Text Embedding Steering (TES), a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process. We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters. Extensive experiments on Stable Diffusion demonstrate that TES outperforms existing training-free baselines in fairness while maintaining competitive image quality. These results highlight that inference-time text embedding optimization is a practical and scalable solution for fairness-aware generation in diffusion models.

Problem

• demographic bias
• diffusion models
• text-to-image generation
• fairness
• stereotypical representations

Innovation

• training-free debiasing
• text embedding steering
• CLIP-guided optimization
• diffusion models
• fairness-aware generation