Efficient bias mitigation in T2I diffusion models using Concept Graphs

2026-07-03

AI Summary

This work addresses the tendency of text-to-image diffusion models to inherit and amplify harmful biases present in their training data, a problem exacerbated by existing debiasing methods that often compromise semantic coherence. To overcome this limitation, the authors propose CO-ALIGN, a novel approach that integrates a concept ontology directly into the diffusion model architecture. By aligning the ontological structures between the text encoder and the denoiser, CO-ALIGN enables three alignment paradigms—encoder-only, denoiser-only, and joint alignment. The method significantly mitigates bias (by 30%) while preserving or even enhancing generation quality, as evidenced by an 11.4-point improvement in ΔFID, a 2.8% gain in fidelity, and an 88% reduction in semantically incoherent outputs. Furthermore, CO-ALIGN improves robustness in downstream tasks such as concept erasure.

Abstract

Text-to-Image diffusion models often propagate harmful bias inherited from the training data. Existing bias mitigation techniques typically intervene only at the text encoder or provide inference-time guidance, often leading to generations that collapse into semantically incoherent outputs. To address these limitations, we introduce CO-ALIGN (Concept Ontology Alignment), a novel bias mitigation approach based on concept-graph alignment that operates on the model's internal concept ontology. By aligning concepts within the text encoder and denoiser, CO-ALIGN achieves substantial bias reduction while preserving generative integrity. We demonstrate the effectiveness of concept-graph alignment across three paradigms: text-encoders, denoisers and joint text-denoiser ontology alignment. CO-ALIGN outperforms the state of the art, improving fairness by $30\%$, $ΔFID=11.4$ in image quality, $2.8\%$ in image fidelity, all while reducing semantically incoherent outputs by $88\%$. Beyond bias mitigation, we show that CO-ALIGN benefits other downstream tasks as well. In particular, our experiments demonstrate that better-aligned internal ontologies enhance concept unlearning robustness across multiple unlearning techniques.

Problem

• bias mitigation
• text-to-image diffusion models
• concept graphs
• semantic coherence
• model bias

Innovation

• Concept Graphs
• Bias Mitigation
• Diffusion Models
• Ontology Alignment
• Text-to-Image Generation