Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentation

2026-06-14

AI Summary

Existing diffusion-based zero-shot segmentation methods are constrained by the trade-off between spatial resolution and contextual information and rely on features extracted from a single, static denoising timestep. This work proposes a context similarity graph construction mechanism that integrates high-resolution attention maps with U-Net encoder features. Leveraging the semantic evolution inherent in the diffusion denoising process—from part-level to object-level representations—it adaptively selects an optimal timestep for each pixel to perform segmentation. The proposed approach significantly outperforms current state-of-the-art methods across multiple zero-shot segmentation benchmarks, demonstrating the effectiveness and superiority of dynamic timestep selection combined with context-aware feature fusion.

Abstract

Zero-shot segmentation has recently shown notable improvement by leveraging the rich visual priors in large-scale text-to-image diffusion models, such as Stable Diffusion. However, current diffusion-based methods often face limitations due to the trade-off between spatial resolution and contextual information, as well as their reliance on a single static timestep for feature extraction. To overcome these challenges, our work introduces two key advancements. First, our Contextual Similarity Maps fuse high-resolution attention maps with rich U-Net encoder features, providing both fine-grained and robust per-pixel representations. Second, we identify an emergent hierarchical semantic progression within the denoising process of various diffusion models: representations transition from part-level abstractions at earlier timesteps to object-level abstractions at later stages. Leveraging this insight, we introduce a mechanism to adaptively select the optimal timestep for each pixel. Extensive experiments demonstrate that our method consistently outperforms existing zero-shot segmentation baselines, validating the efficacy of combining contextual features with dynamic, hierarchical timestep selection.

Problem

• zero-shot segmentation
• diffusion models
• timestep selection
• spatial resolution
• contextual information

Innovation

• zero-shot segmentation
• diffusion models
• adaptive timestep selection
• hierarchical semantics
• contextual similarity maps