DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut

2024-06-05

Neural Information Processing Systems

AI Summary

Unsupervised image segmentation lags significantly behind supervised methods in performance. To address this, we propose DiffCut—a zero-shot semantic segmentation framework leveraging the encoder of a pretrained diffusion-based UNet, requiring no annotated data. Our key insight is the discovery that self-attention features from the final layer of diffusion UNets exhibit strong semantic discriminability, enabling construction of a pixel-wise similarity graph. We further introduce a recursive normalized cut algorithm that softly controls segmentation granularity, balancing object integrity and fine-grained accuracy. DiffCut integrates diffusion feature distillation, spectral clustering, and adaptive graph partitioning. On zero-shot segmentation benchmarks, it substantially outperforms prior state-of-the-art methods, producing high-fidelity segmentations with crisp boundaries and coherent structures. This demonstrates the promise of diffusion models as general-purpose visual foundation encoders.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Foundation models have emerged as powerful tools across various domains including language, vision, and multimodal tasks. While prior works have addressed unsupervised image segmentation, they significantly lag behind supervised models. In this paper, we use a diffusion UNet encoder as a foundation vision encoder and introduce DiffCut, an unsupervised zero-shot segmentation method that solely harnesses the output features from the final self-attention block. Through extensive experimentation, we demonstrate that the utilization of these diffusion features in a graph based segmentation algorithm, significantly outperforms previous state-of-the-art methods on zero-shot segmentation. Specifically, we leverage a recursive Normalized Cut algorithm that softly regulates the granularity of detected objects and produces well-defined segmentation maps that precisely capture intricate image details. Our work highlights the remarkably accurate semantic knowledge embedded within diffusion UNet encoders that could then serve as foundation vision encoders for downstream tasks. Project page at https://diffcut-segmentation.github.io

Problem

• Unsupervised zero-shot semantic segmentation using diffusion features
• Leveraging diffusion UNet encoder for semantic knowledge extraction
• Improving segmentation accuracy with recursive Normalized Cut algorithm

Innovation

• Leverages diffusion UNet encoder features
• Applies recursive Normalized Cut algorithm
• Enables zero-shot semantic segmentation