Exploring Semantic Masked Autoencoder for Self-supervised Point Cloud Understanding

2025-06-27

AI Summary

In self-supervised point cloud learning, existing masked modeling approaches rely on random masking, which fails to capture part-level semantic relationships and thus limits representation generalizability. To address this, we propose the Semantic Masked Autoencoder (SMAE). First, SMAE constructs part-level semantic prototypes to enable interpretable, semantics-guided representation learning. Second, it introduces a semantic-aware masking strategy that prioritizes masking semantically coherent regions to enhance structural understanding. Third, it incorporates a semantic-enhanced prompt fine-tuning mechanism to improve transferability to downstream tasks. Evaluated on ScanObjectNN, ModelNet40, and ShapeNetPart, SMAE achieves significant improvements in both classification and part segmentation performance. Our results demonstrate that explicit semantic modeling—particularly at the part level—is critical for advancing unsupervised point cloud representation learning. The proposed framework bridges the gap between low-level geometric structure and high-level semantic abstraction, offering a principled approach to semantics-aware self-supervision in 3D point clouds.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Segmentation
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Point cloud understanding aims to acquire robust and general feature representations from unlabeled data. Masked point modeling-based methods have recently shown significant performance across various downstream tasks. These pre-training methods rely on random masking strategies to establish the perception of point clouds by restoring corrupted point cloud inputs, which leads to the failure of capturing reasonable semantic relationships by the self-supervised models. To address this issue, we propose Semantic Masked Autoencoder, which comprises two main components: a prototype-based component semantic modeling module and a component semantic-enhanced masking strategy. Specifically, in the component semantic modeling module, we design a component semantic guidance mechanism to direct a set of learnable prototypes in capturing the semantics of different components from objects. Leveraging these prototypes, we develop a component semantic-enhanced masking strategy that addresses the limitations of random masking in effectively covering complete component structures. Furthermore, we introduce a component semantic-enhanced prompt-tuning strategy, which further leverages these prototypes to improve the performance of pre-trained models in downstream tasks. Extensive experiments conducted on datasets such as ScanObjectNN, ModelNet40, and ShapeNetPart demonstrate the effectiveness of our proposed modules.

Problem

• Improving semantic relationships in self-supervised point cloud models
• Addressing limitations of random masking in point cloud understanding
• Enhancing pre-trained model performance in downstream tasks

Innovation

• Prototype-based component semantic modeling module
• Component semantic-enhanced masking strategy
• Component semantic-enhanced prompt-tuning strategy