ArtGS:3D Gaussian Splatting for Interactive Visual-Physical Modeling and Manipulation of Articulated Objects

2025-07-03

AI Summary

Articulated object manipulation remains challenging due to complex kinematics and insufficient physical reasoning capabilities. This paper proposes a vision-physical joint modeling framework based on dynamic 3D Gaussian splatting—the first to introduce differentiable dynamic 3D Gaussian rendering for this task. The method integrates multi-view RGB-D reconstruction, vision-language model–based semantic parsing, and closed-loop physically grounded parameter optimization. It enables cross-morphology adaptive modeling and physically consistent motion control. Evaluated in both simulation and real-world settings, the approach achieves substantial improvements: +23.6% in joint pose estimation accuracy and +31.4% in manipulation success rate. These results demonstrate its effectiveness, generalizability, and scalability across diverse articulated structures.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Articulated object manipulation remains a critical challenge in robotics due to the complex kinematic constraints and the limited physical reasoning of existing methods. In this work, we introduce ArtGS, a novel framework that extends 3D Gaussian Splatting (3DGS) by integrating visual-physical modeling for articulated object understanding and interaction. ArtGS begins with multi-view RGB-D reconstruction, followed by reasoning with a vision-language model (VLM) to extract semantic and structural information, particularly the articulated bones. Through dynamic, differentiable 3DGS-based rendering, ArtGS optimizes the parameters of the articulated bones, ensuring physically consistent motion constraints and enhancing the manipulation policy. By leveraging dynamic Gaussian splatting, cross-embodiment adaptability, and closed-loop optimization, ArtGS establishes a new framework for efficient, scalable, and generalizable articulated object modeling and manipulation. Experiments conducted in both simulation and real-world environments demonstrate that ArtGS significantly outperforms previous methods in joint estimation accuracy and manipulation success rates across a variety of articulated objects. Additional images and videos are available on the project website: https://sites.google.com/view/artgs/home

Problem

• Addresses articulated object manipulation challenges in robotics
• Integrates visual-physical modeling for object understanding
• Enhances joint estimation accuracy and manipulation success

Innovation

• Integrates 3D Gaussian Splatting with visual-physical modeling
• Uses vision-language model for semantic and structural reasoning
• Employs dynamic differentiable rendering for motion optimization