CLIP is All You Need for Human-like Semantic Representations in Stable Diffusion

2025-11-11

AI Summary

This study investigates whether latent diffusion models (e.g., Stable Diffusion) possess human-like semantic understanding in text-to-image generation—specifically, whether their internal representations encode human-interpretable semantic attributes and where such semantics originate. Method: Using linear probing on layer-wise hidden states, we quantitatively assess the decodability of human-annotated semantic attributes (e.g., object category, color, pose), benchmarking against CLIP text embeddings and human judgments. Contribution/Results: We find that semantic representations are predominantly inherited from the CLIP text encoder—not learned or refined during diffusion. The diffusion model functions primarily as a visual decoder; semantic discriminability sharply degrades across reverse diffusion steps. This work provides the first quantitative characterization of semantic division of labor in multimodal generative models, establishing that pre-trained textual encoding—not the diffusion process—is essential for human-aligned semantic understanding. Our findings offer foundational insights for interpretability, controllability, and principled design of generative models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Latent diffusion models such as Stable Diffusion achieve state-of-the-art results on text-to-image generation tasks. However, the extent to which these models have a semantic understanding of the images they generate is not well understood. In this work, we investigate whether the internal representations used by these models during text-to-image generation contain semantic information that is meaningful to humans. To do so, we perform probing on Stable Diffusion with simple regression layers that predict semantic attributes for objects and evaluate these predictions against human annotations. Surprisingly, we find that this success can actually be attributed to the text encoding occurring in CLIP rather than the reverse diffusion process. We demonstrate that groups of specific semantic attributes have markedly different decoding accuracy than the average, and are thus represented to different degrees. Finally, we show that attributes become more difficult to disambiguate from one another during the inverse diffusion process, further demonstrating the strongest semantic representation of object attributes in CLIP. We conclude that the separately trained CLIP vision-language model is what determines the human-like semantic representation, and that the diffusion process instead takes the role of a visual decoder.

Problem

• Investigating semantic understanding in Stable Diffusion's internal representations
• Evaluating how well models decode human-like object attributes
• Determining CLIP's role versus diffusion process in semantic representation

Innovation

• CLIP text encoder provides semantic representations
• Probing reveals semantic attributes in latent space
• Diffusion process acts as visual decoder