Motion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding

2026-08-31

AI Summary

本文提出MoSaiC框架，通过运动显著性互补掩码建模解决点云视频理解问题，结合了CMSM、NFM和CTCP三种方法。

Abstract

Point cloud video representation learning is crucial for 3D dynamic scene understanding. In this paper, we propose MoSaiC, a novel Motion-Saliency Complementary masked modeling framework for self-supervised point cloud video representation learning. MoSaiC couples three components: Curriculum Motion-Saliency Masking (CMSM), which guides the masking process toward motion-salient tokens under a curriculum schedule; Normal-Flow Motion (NFM) modeling, which supervises the local rigid rotation of each token in the Lie algebra so(3) as an explicit geometric motion target; and Cross-view Token Consistency Prediction (CTCP), which enforces consistency between two complementary masked views at the token level. Together, these components allow MoSaiC to effectively capture both appearance and motion dynamics. Extensive experiments on multiple downstream tasks, including action recognition, temporal action segmentation, and point-level semantic segmentation, demonstrate the effectiveness of our approach.

Problem

• Point Cloud Video
• Representation Learning
• 3D Dynamic Scene Understanding

Innovation

• Motion-Saliency Complementary Masked Modeling
• Curriculum Motion-Saliency Masking (CMSM)
• Normal-Flow Motion (NFM) modeling
• Cross-view Token Consistency Prediction (CTCP)