3D sans 3D Scans: Scalable Pre-training from Video-Generated Point Clouds

2025-12-28

AI Summary

This work addresses the critical dependency of 3D representation learning on costly, manually annotated 3D scan data. We propose LAM3C—the first purely video-driven, self-supervised framework for 3D pretraining. LAM3C operates solely on unlabeled videos: it reconstructs point clouds from video frames to model geometric structure, introduces a noise-regularized reconstruction loss to enhance feature robustness, and employs multi-level Laplacian-aware clustering coupled with Sinkhorn-Knopp optimization to enforce geometrically consistent feature learning. Crucially, LAM3C achieves high-performance 3D representation pretraining without any ground-truth 3D supervision—marking the first such result. On indoor semantic and instance segmentation benchmarks, LAM3C consistently outperforms existing self-supervised methods, demonstrating that raw video serves as a scalable, effective, and viable source for large-scale 3D pretraining.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite recent progress in 3D self-supervised learning, collecting large-scale 3D scene scans remains expensive and labor-intensive. In this work, we investigate whether 3D representations can be learned from unlabeled videos recorded without any real 3D sensors. We present Laplacian-Aware Multi-level 3D Clustering with Sinkhorn-Knopp (LAM3C), a self-supervised framework that learns from video-generated point clouds from unlabeled videos. We first introduce RoomTours, a video-generated point cloud dataset constructed by collecting room-walkthrough videos from the web (e.g., real-estate tours) and generating 49,219 scenes using an off-the-shelf feed-forward reconstruction model. We also propose a noise-regularized loss that stabilizes representation learning by enforcing local geometric smoothness and ensuring feature stability under noisy point clouds. Remarkably, without using any real 3D scans, LAM3C achieves higher performance than the previous self-supervised methods on indoor semantic and instance segmentation. These results suggest that unlabeled videos represent an abundant source of data for 3D self-supervised learning.

Problem

• Learning 3D representations from unlabeled videos without real 3D sensors
• Creating a scalable pre-training dataset from video-generated point clouds
• Improving 3D semantic and instance segmentation via self-supervised learning

Innovation

• Self-supervised learning from video-generated point clouds
• Noise-regularized loss for geometric smoothness and stability
• Scalable pre-training without real 3D sensors or scans