ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Models

2025-03-25

AI Summary

To address the inherent ambiguity of visual concepts in single-image understanding, which hinders reliable intrinsic concept extraction, this paper proposes the first unsupervised framework for disentangling intrinsic concepts solely using a pre-trained text-to-image diffusion model. Methodologically, it introduces a two-stage mechanism: (1) leveraging diffusion model gradients to localize interpretable, text-aligned concept regions within the image; and (2) generating semantic masks via cross-modal attention, followed by hierarchical decomposition of concepts into object-level, intrinsic-level, and generic-level categories. The framework requires no annotations, fine-tuning, or auxiliary supervision. It achieves, for the first time, high-precision, fine-grained, and interpretable intrinsic concept extraction from a single image. Extensive evaluations on multiple benchmarks demonstrate significant improvements over state-of-the-art methods. The extracted concepts serve as a robust foundation for downstream controllable image editing and visual reasoning tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

The inherent ambiguity in defining visual concepts poses significant challenges for modern generative models, such as the diffusion-based Text-to-Image (T2I) models, in accurately learning concepts from a single image. Existing methods lack a systematic way to reliably extract the interpretable underlying intrinsic concepts. To address this challenge, we present ICE, short for Intrinsic Concept Extraction, a novel framework that exclusively utilizes a T2I model to automatically and systematically extract intrinsic concepts from a single image. ICE consists of two pivotal stages. In the first stage, ICE devises an automatic concept localization module to pinpoint relevant text-based concepts and their corresponding masks within the image. This critical stage streamlines concept initialization and provides precise guidance for subsequent analysis. The second stage delves deeper into each identified mask, decomposing the object-level concepts into intrinsic concepts and general concepts. This decomposition allows for a more granular and interpretable breakdown of visual elements. Our framework demonstrates superior performance on intrinsic concept extraction from a single image in an unsupervised manner. Project page: https://visual-ai.github.io/ice

Problem

• Extracting interpretable visual concepts from single images
• Overcoming ambiguity in concept definition for generative models
• Decomposing objects into intrinsic and general concepts automatically

Innovation

• Automatic concept localization via T2I model
• Decomposing objects into intrinsic concepts
• Unsupervised single-image concept extraction