Diff-SBSR: Learning Multimodal Feature-Enhanced Diffusion Models for Zero-Shot Sketch-Based 3D Shape Retrieval

2026-04-21

AI Summary

This work addresses the performance bottlenecks in zero-shot sketch-based 3D shape retrieval caused by the absence of category-level supervision and the sparsity of sketches. It introduces, for the first time, a frozen Stable Diffusion model to extract and aggregate discriminative representations from intermediate U-Net layers of both sketches and rendered 3D views. To enhance semantic understanding, the approach further integrates CLIP visual features with text descriptions generated by BLIP, while employing Circle-T loss to dynamically optimize cross-modal alignment. The proposed fine-tuning-free, multimodal feature enhancement strategy effectively bridges the domain gap between sketches and natural images, significantly outperforming existing methods on two public benchmarks and achieving more robust and accurate zero-shot retrieval performance.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper presents the first exploration of text-to-image diffusion models for zero-shot sketch-based 3D shape retrieval (ZS-SBSR). Existing sketch-based 3D shape retrieval methods struggle in zero-shot settings due to the absence of category supervision and the extreme sparsity of sketch inputs. Our key insight is that large-scale pretrained diffusion models inherently exhibit open-vocabulary capability and strong shape bias, making them well suited for zero-shot visual retrieval. We leverage a frozen Stable Diffusion backbone to extract and aggregate discriminative representations from intermediate U-Net layers for both sketches and rendered 3D views. Diffusion models struggle with sketches due to their extreme abstraction and sparsity, compounded by a significant domain gap from natural images. To address this limitation without costly retraining, we introduce a multimodal feature-enhanced strategy that conditions the frozen diffusion backbone with complementary visual and textual cues from CLIP, explicitly enhancing the ability of semantic context capture and concentrating on sketch contours. Specifically, we inject global and local visual features derived from a pretrained CLIP visual encoder, and incorporate enriched textual guidance by combining learnable soft prompts with hard textual descriptions generated by BLIP. Furthermore, we employ the Circle-T loss to dynamically strengthen positive-pair attraction once negative samples are sufficiently separated, thereby adapting to sketch noise and enabling more effective sketch-3D alignment. Extensive experiments on two public benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches in ZS-SBSR.

Problem

• zero-shot
• sketch-based 3D shape retrieval
• extreme sparsity
• domain gap
• category supervision

Innovation

• diffusion models
• zero-shot retrieval
• multimodal feature enhancement
• sketch-based 3D shape retrieval
• CLIP conditioning