PUGS: Zero-shot Physical Understanding with Gaussian Splatting

2025-02-17

AI Summary

Existing robots struggle to infer physical attributes—such as mass, friction, and hardness—of objects without prior knowledge. This paper introduces 3D Gaussian Splatting to physical attribute understanding for the first time, proposing a zero-shot, annotation-free prediction framework. It reconstructs object geometry and appearance via Gaussian rasterization; enhances reconstruction robustness through a geometry-aware regularization loss and a region-aware contrastive loss; and enables cross-regional physical reasoning via a feature-driven attribute propagation mechanism and a Gaussian-adapted volumetric integration module. Evaluated on the ABO-500 mass prediction benchmark, our method achieves state-of-the-art performance. Moreover, it significantly improves grasp success rates in real-world manipulation tasks. All code, datasets, and models are publicly released.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Low Level & Physics-based Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Current robotic systems can understand the categories and poses of objects well. But understanding physical properties like mass, friction, and hardness, in the wild, remains challenging. We propose a new method that reconstructs 3D objects using the Gaussian splatting representation and predicts various physical properties in a zero-shot manner. We propose two techniques during the reconstruction phase: a geometry-aware regularization loss function to improve the shape quality and a region-aware feature contrastive loss function to promote region affinity. Two other new techniques are designed during inference: a feature-based property propagation module and a volume integration module tailored for the Gaussian representation. Our framework is named as zero-shot physical understanding with Gaussian splatting, or PUGS. PUGS achieves new state-of-the-art results on the standard benchmark of ABO-500 mass prediction. We provide extensive quantitative ablations and qualitative visualization to demonstrate the mechanism of our designs. We show the proposed methodology can help address challenging real-world grasping tasks. Our codes, data, and models are available at https://github.com/EverNorif/PUGS

Problem

• Predicts physical properties zero-shot
• Improves 3D object reconstruction quality
• Enhances real-world robotic grasping tasks

Innovation

• Gaussian splatting for 3D reconstruction
• Geometry-aware regularization loss function
• Feature-based property propagation module