Problem
Research questions and friction points this paper is trying to address.
open vocabulary
3D semantic segmentation
annotation-free
language-space embedding
out-of-vocabulary
GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting
🤖 AI Summary
本文提出GoDeep方法，通过将图像描述投影到语言空间进行3D场景理解，无需3D标注和编码器，解决了开放词汇3D语义分割问题。
📝 Abstract
Open vocabulary 3D semantic segmentation methods typically lift CLIP features into 3D. This embeds points in a joint vision-language space known to behave like a bag-of-words on compositional tasks. Furthermore, even annotation free variants often require a large 3D training corpus and a dedicated 3D encoder per domain. Instead we use a vision-language model purely as a translator. It produces structured, entity-level descriptions of each posed image. These descriptions are grounded, projected, and aggregated directly in a general-purpose, language-only embedding space, with no 3D training corpus or encoder required. On ScanNet++, our pipeline is competitive with strong annotation free baselines trained on ScanNet. On a 5-building cultural heritage benchmark, raw scores initially favor a CLIP-based variant, but a single systematic vocabulary correction reverses this ranking. An effect confirmed by a second, independent correction on a different class, indicating that language-space embeddings track physical content more faithfully. This fidelity extends to genuinely out-of-vocabulary (OOV) objects on ScanNet++ proving that language-space embeddings separate presence from absence objects far more sharply than CLIP-based embeddings do. GoDeep also localize these OOV objects within the scene, all without any 2D-3D annotation. Because every representation remains discrete text, predictions are also explainable at the point level. Finally, exploiting both a heuristic weighting, that favors precise over merely frequent observations and GoDeep's explainability property, we propose an aggregation strategy, as a proof of concept, that favors finer elements localization.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Annotation-Free
Open-Vocabulary
Language-Space Embedding
Explainability
🔎 Similar Papers
