Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation

2025-09-29

AI Summary

This work addresses the problem of counterfactual generation with causal inconsistency in text-to-image diffusion models. To enable faithful, causally grounded attribute intervention while keeping the pre-trained model frozen, we propose a modular adaptation framework. Methodologically, we construct a structured causal model to explicitly encode inter-attribute causal dependencies; design a prompt-aligned injection mechanism for precise targeting of intervention goals; and introduce a conditional token contrastive loss alongside dual-attribute regularization to jointly enhance semantic controllability and attribute disentanglement, thereby mitigating spurious correlations. Experiments on both synthetic (Pendulum) and real-world (ADNI) datasets demonstrate significant improvements over state-of-the-art methods: a 91% reduction in attribute control MAE, an 87% improvement in MRI generation FID, and strict preservation of image identity consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Causal Learning
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present Causal-Adapter, a modular framework that adapts frozen text-to-image diffusion backbones for counterfactual image generation. Our method enables causal interventions on target attributes, consistently propagating their effects to causal dependents without altering the core identity of the image. In contrast to prior approaches that rely on prompt engineering without explicit causal structure, Causal-Adapter leverages structural causal modeling augmented with two attribute regularization strategies: prompt-aligned injection, which aligns causal attributes with textual embeddings for precise semantic control, and a conditioned token contrastive loss to disentangle attribute factors and reduce spurious correlations. Causal-Adapter achieves state-of-the-art performance on both synthetic and real-world datasets, with up to 91% MAE reduction on Pendulum for accurate attribute control and 87% FID reduction on ADNI for high-fidelity MRI image generation. These results show that our approach enables robust, generalizable counterfactual editing with faithful attribute modification and strong identity preservation.

Problem

• Adapting diffusion models for counterfactual image generation
• Enabling causal interventions on attributes with identity preservation
• Reducing spurious correlations through structural causal modeling

Innovation

• Modular framework adapting frozen diffusion backbones
• Structural causal modeling with attribute regularization strategies
• Prompt-aligned injection and token contrastive loss