Causally Steered Diffusion for Automated Video Counterfactual Generation

2025-06-17

AI Summary

To address the causal integrity degradation commonly observed in video counterfactual editing, this paper proposes a black-box, fine-tuning-free causal-aware diffusion framework. Methodologically, it introduces the first causal-graph-driven text prompt optimization paradigm, synergizing vision-language model reasoning with latent-space prompt engineering to steer pre-trained latent diffusion models (LDMs) toward generating causally coherent “what-if” videos—without requiring access to the target video editing model’s architecture or parameters. The framework jointly optimizes counterfactual causal faithfulness and minimal intervention, ensuring compatibility with arbitrary off-the-shelf video editing systems. Experiments demonstrate significant improvements over baselines across both perceptual quality metrics (FVD, LPIPS) and counterfactual-specific metrics (Causal Faithfulness, Intervention Consistency). To our knowledge, this is the first work achieving high-fidelity, causally plausible counterfactual video generation solely via prompt engineering within the native LDM distribution.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Adapting text-to-image (T2I) latent diffusion models for video editing has shown strong visual fidelity and controllability, but challenges remain in maintaining causal relationships in video content. Edits affecting causally dependent attributes risk generating unrealistic or misleading outcomes if these relationships are ignored. In this work, we propose a causally faithful framework for counterfactual video generation, guided by a vision-language model (VLM). Our method is agnostic to the underlying video editing system and does not require access to its internal mechanisms or finetuning. Instead, we guide the generation by optimizing text prompts based on an assumed causal graph, addressing the challenge of latent space control in LDMs. We evaluate our approach using standard video quality metrics and counterfactual-specific criteria, such as causal effectiveness and minimality. Our results demonstrate that causally faithful video counterfactuals can be effectively generated within the learned distribution of LDMs through prompt-based causal steering. With its compatibility with any black-box video editing system, our method holds significant potential for generating realistic"what-if"video scenarios in diverse areas such as healthcare and digital media.

Problem

• Maintaining causal relationships in video counterfactual generation
• Avoiding unrealistic outcomes from edits on dependent attributes
• Controlling latent space in diffusion models via causal prompts

Innovation

• Causally faithful framework for video counterfactuals
• Vision-language model guides prompt optimization
• Compatible with black-box video editing systems