Leveraging Foundation Models for Causal Generative Modeling

2026-05-22

AI Summary

This work addresses the absence of a unified framework for leveraging the zero-shot reasoning capabilities of pretrained foundation models in end-to-end visual causal inference. To bridge this gap, the authors propose FM-CGM, a modular architecture that integrates large language reasoning models with text-to-image diffusion models for the first time. The framework comprises three components—concept extractor, manipulator, and counterfactual generator—to jointly perform zero-shot causal discovery, intervention, and counterfactual image generation. Its core innovation lies in the Causal Semantic Guidance (CSG) mechanism, which employs cross-attention during interventions to preserve non-relevant image regions while precisely propagating semantic changes to descendant concepts. Experimental results demonstrate that the method effectively identifies plausible causal structures and produces high-fidelity counterfactual images.

Abstract

Causal generative modeling is essential for developing reliable and transparent AI systems capable of counterfactual reasoning. While existing approaches focus on integrating causal constraints during the training of generative models, they often lack a unified framework to leverage the zero-shot reasoning capabilities of pretrained foundation models. We introduce FM-CGM, a modular framework for end-to-end visual causal reasoning using pretrained foundation models. FM-CGM formalizes the causal pipeline through three core components: a concept extractor, a concept manipulator, and a counterfactual generator. By leveraging a large reasoning model for causal inference and a text-to-image diffusion model for generation, our approach enables zero-shot causal discovery, intervention, and counterfactual generation. We then develop Causal Semantic Guidance (CSG), a cross-attention-based mechanism that ensures semantic interventions propagate to descendant concepts while preserving invariant regions. We empirically show that our approach can identify plausible causal structures and is suitable for faithful counterfactual image generation.

Problem

• causal generative modeling
• foundation models
• counterfactual reasoning
• zero-shot reasoning
• visual causal reasoning

Innovation

• foundation models
• causal generative modeling
• counterfactual generation
• zero-shot reasoning
• Causal Semantic Guidance