EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation

2026-07-01

AI Summary

This work addresses the pervasive issue of demographic biases—particularly gender bias—in text-to-image diffusion models when prompted with neutral inputs. The authors propose a training-free, inference-time intervention that dynamically distinguishes between neutral and attribute-laden prompts via a gating mechanism. For neutral prompts, implicit demographic attributes are removed and replaced with user-specified target attributes, while attribute-relevant prompts remain unaltered. This approach enables sample-level, cross-model fairness control by leveraging precomputed contrastive prompt steering vectors to guide generation in the cross-attention layers of diverse models, including Stable Diffusion 1.5, 2.1, XL, and SANA. Experimental results demonstrate up to an 87% reduction in gender imbalance gap, with negligible degradation in image fidelity or text-image alignment.

Abstract

Text-to-image diffusion models power everyday creative tasks, but they still reproduce the demographic biases in their training data. On common prompts such as ``a photo of a nurse,'' ``a photo of a CEO'', they skew their outputs toward one gender, driven by the statistics of training data rather than anything in the text. Existing debiasing methods show promise in narrow settings but require retraining, batch-level control, or prompt-specific tuning, limiting their scalability. We propose \emph{EquiSteer}, a training-free method that works per sample by steering cross-attention (CA) activations at inference time. For each target attribute, EquiSteer precomputes steering vectors from contrastive prompts. Then at generation time, a prompt-aware gate leaves attribute-specific prompts untouched, while for neutral ones it clears existing attribute signals from the CA activations and injects a target attribute. Across SD-1.5, SD-2.1, SDXL, and SANA, EquiSteer reduces the average parity gap by up to $87\%$, with minimal effect on image quality and text-image alignment. Code is available at \href{https://github.com/Atmyre/EquiSteer}{https://github.com/Atmyre/EquiSteer}.%

Problem

• demographic bias
• text-to-image generation
• fairness
• diffusion models
• gender skew

Innovation

• bias mitigation
• cross-attention steering
• text-to-image generation
• training-free debiasing
• fairness in AI