Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?

2025-02-07

AI Summary

Diffusion models (DMs) exhibit systematic failure in modeling fine-grained causal regularities among image features—e.g., the geometric relationship between solar altitude and shadow length—despite their strong generative capabilities. Method: We introduce a synthetic, controllable task framework to rigorously assess rule learning in DMs. Using theoretical analysis, we prove an inherent incompatibility between the denoising score-matching objective and exact rule consistency, deriving a fundamental constant lower bound on rule-learning error. We further analyze classifier-guided sampling and establish its intrinsic limitations for capturing such fine-grained dependencies. Results: Empirical evaluation across four strongly correlated feature tasks—including solar geometry, perspective scaling, lighting consistency, and occlusion ordering—reveals consistent failure of state-of-the-art models (e.g., Stable Diffusion 3.5). This work provides the first formal characterization of DMs’ inductive bias toward causal structure, offering both theoretical insights and a benchmark for evaluating causal representational capacity in generative models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite the remarkable success of diffusion models (DMs) in data generation, they exhibit specific failure cases with unsatisfactory outputs. We focus on one such limitation: the ability of DMs to learn hidden rules between image features. Specifically, for image data with dependent features ($mathbf{x}$) and ($mathbf{y}$) (e.g., the height of the sun ($mathbf{x}$) and the length of the shadow ($mathbf{y}$)), we investigate whether DMs can accurately capture the inter-feature rule ($p(mathbf{y}|mathbf{x})$). Empirical evaluations on mainstream DMs (e.g., Stable Diffusion 3.5) reveal consistent failures, such as inconsistent lighting-shadow relationships and mismatched object-mirror reflections. Inspired by these findings, we design four synthetic tasks with strongly correlated features to assess DMs' rule-learning abilities. Extensive experiments show that while DMs can identify coarse-grained rules, they struggle with fine-grained ones. Our theoretical analysis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity. To mitigate this, we introduce a common technique - incorporating additional classifier guidance during sampling, which achieves (limited) improvements. Our analysis reveals that the subtle signals of fine-grained rules are challenging for the classifier to capture, providing insights for future exploration.

Problem

• Assess DMs' ability to learn hidden feature rules.
• Identify limitations in DMs' fine-grained rule learning.
• Improve DMs' performance using classifier guidance.

Innovation

• Assesses DMs' rule-learning abilities
• Introduces classifier guidance technique
• Highlights DSM's constant rule errors