Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales

2026-05-27

AI Summary

Existing debiasing methods struggle to maintain group fairness under varying guidance scales. This work is the first to decompose total bias into model bias and guidance bias, revealing that the latter monotonically increases with guidance scale and dominates unfairness in high-guidance regimes. Building on this insight, the authors propose StayFair, a plug-in algorithm compatible with existing fair generative models. StayFair preserves demographic proportions across guidance scales by either balancing output distributions in classifier guidance or introducing prompt-dependent null-embedding offsets in classifier-free guidance. Experiments demonstrate that StayFair effectively enforces strong demographic parity across diverse guidance scales in both class-conditional and text-to-image generation tasks, without compromising sample quality.

Abstract

Diffusion models steer conditional generation with a tunable guidance scale to trade off prompt alignment and diversity. However, existing debiasing techniques are optimized for a single scale, degrading fairness when users adjust this parameter. We trace this behavior to a previously overlooked source by decomposing total bias into two components: a model bias and a guidance bias. While prior work primarily targets the former, we show that the guidance bias grows monotonically with the guidance scale, eventually dominating the high-guidance regimes users prefer. To address this, we extend Strong Demographic Parity to guidance and derive a condition under which the target distribution retains its group ratio across guidance scales. We propose StayFair, which leverages this condition to design fair guidance algorithms in both regimes. For classifier guidance, it equalizes the classifier's output distributions across groups; for classifier-free guidance, it shifts the null embedding by a prompt-dependent offset. Because StayFair modifies only the guidance step, it is orthogonal to model debiasing and can be layered onto existing fair diffusion models to extend their fairness across guidance scales. Across class-conditional and text-to-image generation, StayFair decouples fairness from the guidance scale without sacrificing image quality.

Problem

• diffusion models
• group fairness
• guidance scale
• bias
• fairness degradation

Innovation

• guidance bias
• group fairness
• diffusion models
• Strong Demographic Parity
• classifier-free guidance