Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning

2026-06-23

AI Summary

This study investigates whether text-to-image (T2I) generation models possess genuine causal reasoning capabilities or merely rely on statistical visual-linguistic associations. To this end, the authors introduce CF-World, the first counterfactual benchmark for T2I models, featuring a three-tiered task hierarchy designed to evaluate generation performance under conditions that violate real-world priors. They also propose CF-Eval, an automated evaluation framework based on vision-language models (VLMs), and define two novel metrics—Prior Resistance Rate and Reasoning Retention Rate—to systematically quantify a model’s ability to resist ingrained commonsense priors and retain causal reasoning. Experimental results demonstrate that state-of-the-art T2I models exhibit significant performance degradation in counterfactual scenarios, revealing their strong dependence on co-occurrence patterns in training data and a lack of true causal understanding.

Abstract

Text-to-image (T2I) generation models have achieved remarkable progress in producing visually realistic images from natural language prompts. Yet it remains unclear whether their success reflects genuine causal understanding or sophisticated pattern matching over visual-textual correlations. Inspired by Russell's inductivist turkey, we introduce Counterfactual-World (CF-World), a counterfactual benchmark designed to investigate whether text-to-image models can generate images under rules that systematically contradict real-world priors. CF-World organizes each scenario into three progressive levels: factual generation under ordinary world knowledge, explicit counterfactual generation with direct visual instructions, and implicit counterfactual generation requiring causal deduction from altered rules. We evaluate both open-source and closed-source T2I models using a Vision Language Model (VLM)-based evaluator (CF-Eval). Furthermore, we introduce two metrics: Prior Resistance Rate (PRR), which measures a model's ability to overcome entrenched real-world priors, and Reasoning Retention Rate (RRR), which assesses whether models can maintain reasoning-dependent counterfactual generation without explicit visual cues. Experiments show that all models exhibit sharp degradation from factual to counterfactual settings. Further analyses suggest that these failures arise because current T2I models encode world knowledge and visual appearances as tightly coupled patterns. Consequently, their heavy reliance on frequent visual co-occurrences within the training data forces them to default to familiar commonsense priors when tasked with rendering counterfactual worlds.

Problem

• text-to-image generation
• causal reasoning
• counterfactual reasoning
• inductive bias
• commonsense priors

Innovation

• counterfactual reasoning
• text-to-image generation
• causal understanding
• benchmarking
• prior resistance