3D Part Segmentation via Geometric Aggregation of 2D Visual Features

2024-12-05

AI Summary

Existing 3D part segmentation methods exhibit poor generalization in real-world complex scenes, struggling to simultaneously ensure multi-view robustness and effective geometric structure modeling; direct adaptation of vision-language models (VLMs) is hindered by heavy reliance on prompt engineering and insufficient exploitation of 3D geometric information. Method: We propose a zero-shot, open-vocabulary 3D part segmentation framework: multi-view rendering extracts CLIP visual features, which are jointly aligned with explicit 3D geometry via a differentiable 3D–2D–3D projection; a geometry-aware feature aggregation mechanism employs spatially consistent, geometry-weighted attention to fuse multi-view features; finally, spectral clustering coupled with semantic label alignment enables category-agnostic part discovery. Contribution/Results: Our method eliminates prompt engineering and achieves zero-shot state-of-the-art performance across five synthetic and real-world datasets—covering textureless/colored and rigid/non-rigid objects—demonstrating high efficiency and strong cross-domain and cross-deformation generalization.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Supervised 3D part segmentation models are tailored for a fixed set of objects and parts, limiting their transferability to open-set, real-world scenarios. Recent works have explored vision-language models (VLMs) as a promising alternative, using multi-view rendering and textual prompting to identify object parts. However, naively applying VLMs in this context introduces several drawbacks, such as the need for meticulous prompt engineering, and fails to leverage the 3D geometric structure of objects. To address these limitations, we propose COPS, a COmprehensive model for Parts Segmentation that blends the semantics extracted from visual concepts and 3D geometry to effectively identify object parts. COPS renders a point cloud from multiple viewpoints, extracts 2D features, projects them back to 3D, and uses a novel geometric-aware feature aggregation procedure to ensure spatial and semantic consistency. Finally, it clusters points into parts and labels them. We demonstrate that COPS is efficient, scalable, and achieves zero-shot state-of-the-art performance across five datasets, covering synthetic and real-world data, texture-less and coloured objects, as well as rigid and non-rigid shapes. The code is available at https://3d-cops.github.io.

Problem

• 3D part segmentation
• VLM limitations
• stereo shape information

Innovation

• COPS Model
• Multi-view Object Recognition
• 3D Part Segmentation