Sonata: Self-Supervised Learning of Reliable Point Representations

2025-03-20

AI Summary

This paper addresses the collapse of 3D self-supervised point cloud representations into low-level geometric features—termed “geometric shortcuts”—which severely degrades linear probe performance. To tackle this, we propose Sonata: the first framework to systematically identify and mitigate geometric shortcuts. Sonata breaks such shortcuts via spatial information masking (e.g., coordinate perturbation or dropout), enforces semantic consistency through feature centering constraints, and strengthens semantic dependencies via self-distillation. It introduces the first self-distillation architecture for point clouds, trained on a large-scale dataset of 140K scans. On ScanNet, Sonata boosts linear probe accuracy from 21.8% to 72.5%. With only 1% labeled data, it matches the performance of fully supervised state-of-the-art methods. Moreover, it demonstrates strong zero-shot semantic grouping and spatial reasoning capabilities, significantly advancing the performance frontier for indoor and outdoor 3D perception.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Search and Optimization: Learning to Search
• Computer Vision: 3D Computer Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

In this paper, we question whether we have a reliable self-supervised point cloud model that can be used for diverse 3D tasks via simple linear probing, even with limited data and minimal computation. We find that existing 3D self-supervised learning approaches fall short when evaluated on representation quality through linear probing. We hypothesize that this is due to what we term the"geometric shortcut", which causes representations to collapse to low-level spatial features. This challenge is unique to 3D and arises from the sparse nature of point cloud data. We address it through two key strategies: obscuring spatial information and enhancing the reliance on input features, ultimately composing a Sonata of 140k point clouds through self-distillation. Sonata is simple and intuitive, yet its learned representations are strong and reliable: zero-shot visualizations demonstrate semantic grouping, alongside strong spatial reasoning through nearest-neighbor relationships. Sonata demonstrates exceptional parameter and data efficiency, tripling linear probing accuracy (from 21.8% to 72.5%) on ScanNet and nearly doubling performance with only 1% of the data compared to previous approaches. Full fine-tuning further advances SOTA across both 3D indoor and outdoor perception tasks.

Problem

• Reliable self-supervised point cloud model for 3D tasks
• Overcoming geometric shortcut in 3D self-supervised learning
• Improving representation quality with limited data and computation

Innovation

• Self-supervised learning for 3D point clouds
• Obscuring spatial information to prevent collapse
• Self-distillation with 140k point clouds