SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting

2025-06-04

AI Summary

This work addresses the challenging problem of articulated object reconstruction and kinematic inference without 3D supervision. We propose the first fully self-supervised articulated 3D Gaussian Splatting method, requiring only two RGB images of an object in distinct poses. Our approach enables part-level 3D reconstruction, kinematic parameter estimation, and real-time novel-view/novel-pose rendering. Key contributions include: (i) introducing differentiable motion parameters for each 3D Gaussian to establish geometric self-supervision; (ii) designing a multi-stage joint optimization framework that simultaneously improves part segmentation, reconstruction robustness, and kinematic plausibility; and (iii) incorporating a part-aware density control mechanism. Evaluated on multiple benchmarks and real-world handheld sequences, our method achieves state-of-the-art performance, supports rasterization-based rendering at over 30 FPS, and significantly enhances reconstruction accuracy and cross-pose generalization for complex articulated structures.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: State Estimation
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reconstructing articulated objects prevalent in daily environments is crucial for applications in augmented/virtual reality and robotics. However, existing methods face scalability limitations (requiring 3D supervision or costly annotations), robustness issues (being susceptible to local optima), and rendering shortcomings (lacking speed or photorealism). We introduce SplArt, a self-supervised, category-agnostic framework that leverages 3D Gaussian Splatting (3DGS) to reconstruct articulated objects and infer kinematics from two sets of posed RGB images captured at different articulation states, enabling real-time photorealistic rendering for novel viewpoints and articulations. SplArt augments 3DGS with a differentiable mobility parameter per Gaussian, achieving refined part segmentation. A multi-stage optimization strategy is employed to progressively handle reconstruction, part segmentation, and articulation estimation, significantly enhancing robustness and accuracy. SplArt exploits geometric self-supervision, effectively addressing challenging scenarios without requiring 3D annotations or category-specific priors. Evaluations on established and newly proposed benchmarks, along with applications to real-world scenarios using a handheld RGB camera, demonstrate SplArt's state-of-the-art performance and real-world practicality. Code is publicly available at https://github.com/ripl/splart.

Problem

• Reconstructing articulated objects without 3D supervision
• Improving robustness and accuracy in part segmentation
• Enabling real-time photorealistic rendering for novel articulations

Innovation

• Uses 3D Gaussian Splatting for reconstruction
• Self-supervised, category-agnostic framework
• Multi-stage optimization for robustness