Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model

2025-03-20

AI Summary

This work addresses generalized few-shot 3D point cloud segmentation—specifically, achieving high-accuracy segmentation for novel classes using only a handful of annotated samples, while preserving performance on base classes. To tackle the challenge of leveraging dense yet noisy pseudo-labels generated by 3D vision-language models (VLMs), we propose three synergistic mechanisms: prototype-guided pseudo-label filtering, adaptive context filling, and mixed old-and-new class embedding—jointly enhancing generalization and stability. We introduce two new benchmarks featuring high diversity and cross-scene generalizability, addressing critical limitations in existing evaluation protocols. Extensive experiments across multiple models and datasets demonstrate an average +8.2% mIoU gain on novel classes, with zero degradation in base-class performance. Our code is publicly available.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Generalized few-shot 3D point cloud segmentation (GFS-PCS) adapts models to new classes with few support samples while retaining base class segmentation. Existing GFS-PCS methods enhance prototypes via interacting with support or query features but remain limited by sparse knowledge from few-shot samples. Meanwhile, 3D vision-language models (3D VLMs), generalizing across open-world novel classes, contain rich but noisy novel class knowledge. In this work, we introduce a GFS-PCS framework that synergizes dense but noisy pseudo-labels from 3D VLMs with precise yet sparse few-shot samples to maximize the strengths of both, named GFS-VL. Specifically, we present a prototype-guided pseudo-label selection to filter low-quality regions, followed by an adaptive infilling strategy that combines knowledge from pseudo-label contexts and few-shot samples to adaptively label the filtered, unlabeled areas. Additionally, we design a novel-base mix strategy to embed few-shot samples into training scenes, preserving essential context for improved novel class learning. Moreover, recognizing the limited diversity in current GFS-PCS benchmarks, we introduce two challenging benchmarks with diverse novel classes for comprehensive generalization evaluation. Experiments validate the effectiveness of our framework across models and datasets. Our approach and benchmarks provide a solid foundation for advancing GFS-PCS in the real world. The code is at https://github.com/ZhaochongAn/GFS-VL

Problem

• Adapts 3D point cloud segmentation to new classes with few samples.
• Combines dense pseudo-labels from 3D VLMs with sparse few-shot samples.
• Introduces challenging benchmarks for comprehensive generalization evaluation.

Innovation

• Combines 3D VLMs with few-shot samples
• Uses prototype-guided pseudo-label selection
• Introduces novel-base mix training strategy