DINO Eats CLIP: Adapting Beyond Knowns for Open-set 3D Object Retrieval

2026-04-21

AI Summary

This work addresses the limitations of existing methods in open-set 3D object retrieval, which often rely solely on CLIP encoders and consequently suffer from insufficient fine-grained features and restricted generalization. To overcome these issues, the authors propose the DEC framework, which integrates a frozen DINO self-supervised encoder with the CLIP semantic space. DEC introduces a Chunking and Adapting module to dynamically aggregate multi-view local features and incorporates a Virtual Feature Synthesis mechanism to explicitly generate virtual features tailored for unseen categories, thereby mitigating bias toward known classes. Experimental results demonstrate that DEC significantly outperforms current state-of-the-art approaches on standard open-set 3D retrieval benchmarks, achieving notably enhanced discriminability and generalization for unknown categories.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Vision foundation models have shown great promise for open-set 3D object retrieval (3DOR) through efficient adaptation to multi-view images. Leveraging semantically aligned latent space, previous work typically adapts the CLIP encoder to build view-based 3D descriptors. Despite CLIP's strong generalization ability, its lack of fine-grainedness prompted us to explore the potential of a more recent self-supervised encoder-DINO. To address this, we propose DINO Eats CLIP (DEC), a novel framework for dynamic multi-view integration that is regularized by synthesizing data for unseen classes. We first find that simply mean-pooling over view features from a frozen DINO backbone gives decent performance. Yet, further adaptation causes severe overfitting on average view patterns of known classes. To combat it, we then design a module named Chunking and Adapting Module (CAM). It segments multi-view images into chunks and dynamically integrates local view relations, yielding more robust features than the standard pooling strategy. Finally, we propose Virtual Feature Synthesis (VFS) module to mitigate bias towards known categories explicitly. Under the hood, VFS leverages CLIP's broad, pre-aligned vision-language space to synthesize virtual features for unseen classes. By exposing DEC to these virtual features, we greatly enhance its open-set discrimination capacity. Extensive experiments on standard open-set 3DOR benchmarks demonstrate its superior efficacy.

Problem

• open-set 3D object retrieval
• vision foundation models
• unknown class generalization
• multi-view integration
• feature adaptation

Innovation

• DINO
• CLIP
• open-set 3D object retrieval
• multi-view integration
• virtual feature synthesis