ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their Self-Supervised Pretraining

2024-08-20

arXiv.org

AI Summary

This work addresses the problem of direct 3D understanding in the native space of 3D Gaussian Splatting (3DGS). To this end, we introduce ShapeSplat—the first large-scale 3DGS dataset comprising 65K objects across 87 categories—and propose the Gaussian Masked Autoencoder (Gaussian-MAE) for unsupervised pretraining. Key contributions include: (i) the first empirical revelation of significant distributional shift between optimized Gaussian centroids and their initial point-cloud counterparts; and (ii) the design of Gaussian feature grouping and splat pooling layers, enabling structured, task-adaptive embedding of Gaussian parameters. On downstream classification and segmentation tasks, our approach achieves substantial fine-tuning performance gains. Comprehensive experiments further validate the intrinsic representational capacity of Gaussian parameters, advancing 3DGS from a rendering-centric paradigm toward native 3D understanding.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: 3D Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

3D Gaussian Splatting (3DGS) has become the de facto method of 3D representation in many vision tasks. This calls for the 3D understanding directly in this representation space. To facilitate the research in this direction, we first build a large-scale dataset of 3DGS using the commonly used ShapeNet and ModelNet datasets. Our dataset ShapeSplat consists of 65K objects from 87 unique categories, whose labels are in accordance with the respective datasets. The creation of this dataset utilized the compute equivalent of 2 GPU years on a TITAN XP GPU. We utilize our dataset for unsupervised pretraining and supervised finetuning for classification and segmentation tasks. To this end, we introduce extbf{ extit{Gaussian-MAE}}, which highlights the unique benefits of representation learning from Gaussian parameters. Through exhaustive experiments, we provide several valuable insights. In particular, we show that (1) the distribution of the optimized GS centroids significantly differs from the uniformly sampled point cloud (used for initialization) counterpart; (2) this change in distribution results in degradation in classification but improvement in segmentation tasks when using only the centroids; (3) to leverage additional Gaussian parameters, we propose Gaussian feature grouping in a normalized feature space, along with splats pooling layer, offering a tailored solution to effectively group and embed similar Gaussians, which leads to notable improvement in finetuning tasks.

Problem

• Builds a large-scale 3D Gaussian Splatting dataset for representation learning
• Enables unsupervised pretraining and supervised finetuning for 3D tasks
• Proposes methods to effectively leverage Gaussian parameters for classification and segmentation

Innovation

• Large-scale 3DGS dataset creation from ShapeNet
• Gaussian-MAE for unsupervised pretraining Gaussian parameters
• Gaussian feature grouping with splats pooling layer