Decoupled Classifier-Free Guidance for Counterfactual Diffusion Models

2025-06-17

AI Summary

Standard classifier-free guidance (CFG) employs a uniform global guidance weight for counterfactual image generation under causal interventions, leading to identity distortion and amplification of non-target attributes. Method: We propose Decoupled Classifier-Free Guidance (DCFG), the first CFG variant that partitions conditional variables into *intervention* and *invariant* groups based on a causal graph, enabling group-wise, differentiated guidance. DCFG supports user-defined semantic grouping and reversible interventions, integrating causal modeling, group-level conditional control, and a model-agnostic guidance decoupling mechanism within diffusion models via DDIM inversion and attribute-split embeddings. Results: Evaluated on CelebA-HQ, MIMIC-CXR, and EMBED, DCFG significantly improves intervention fidelity, suppresses spurious attribute changes, and enhances both reversibility and interpretability of generated counterfactuals.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Counterfactual image generation aims to simulate realistic visual outcomes under specific causal interventions. Diffusion models have recently emerged as a powerful tool for this task, combining DDIM inversion with conditional generation via classifier-free guidance (CFG). However, standard CFG applies a single global weight across all conditioning variables, which can lead to poor identity preservation and spurious attribute changes - a phenomenon known as attribute amplification. To address this, we propose Decoupled Classifier-Free Guidance (DCFG), a flexible and model-agnostic framework that introduces group-wise conditioning control. DCFG builds on an attribute-split embedding strategy that disentangles semantic inputs, enabling selective guidance on user-defined attribute groups. For counterfactual generation, we partition attributes into intervened and invariant sets based on a causal graph and apply distinct guidance to each. Experiments on CelebA-HQ, MIMIC-CXR, and EMBED show that DCFG improves intervention fidelity, mitigates unintended changes, and enhances reversibility, enabling more faithful and interpretable counterfactual image generation.

Problem

• Improves counterfactual image generation fidelity
• Mitigates unintended attribute changes in diffusion models
• Enhances identity preservation via selective guidance

Innovation

• Decoupled Classifier-Free Guidance for control
• Attribute-split embedding disentangles semantic inputs
• Group-wise conditioning based on causal graph