GSDeformer: Direct, Real-time and Extensible Cage-based Deformation for 3D Gaussian Splatting

2024-05-24

AI Summary

3D Gaussian Splatting (3DGS) lacks native support for real-time, scalable cage-based deformation. Method: We propose a lightweight, architecture-preserving deformation framework that introduces a proxy point cloud as an intermediary: (i) automatically constructing a deformation cage adapted to the 3DGS geometry distribution; (ii) precisely mapping cage-driven point cloud deformations to Gaussian positions, covariances, and opacities; and (iii) modeling bending effects via adaptive Gaussian splitting. Contribution/Results: Our method requires no retraining or fine-tuning and is plug-and-play across diverse 3DGS variants. It achieves >30 FPS real-time rendering while significantly improving geometric fidelity under extreme deformations and enhancing cross-scene generalization. To our knowledge, this is the first approach enabling high-quality, zero-intrusion cage-controlled deformation of 3DGS.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We present GSDeformer, a method that enables cage-based deformation on 3D Gaussian Splatting (3DGS). Our approach bridges cage-based deformation and 3DGS by using a proxy point-cloud representation. This point cloud is generated from 3D Gaussians, and deformations applied to the point cloud are translated into transformations on the 3D Gaussians. To handle potential bending caused by deformation, we incorporate a splitting process to approximate it. Our method does not modify or extend the core architecture of 3D Gaussian Splatting, making it compatible with any trained vanilla 3DGS or its variants. Additionally, we automate cage construction for 3DGS and its variants using a render-and-reconstruct approach. Experiments demonstrate that GSDeformer delivers superior deformation results compared to existing methods, is robust under extreme deformations, requires no retraining for editing, runs in real-time, and can be extended to other 3DGS variants. Project Page: https://jhuangbu.github.io/gsdeformer/

Problem

• Enables cage-based deformation on 3D Gaussian Splatting
• Automates cage construction for 3DGS using render-and-reconstruct
• Ensures real-time deformation without retraining or architecture changes

Innovation

• Uses proxy point-cloud for cage-based deformation
• Incorporates splitting to handle bending effects
• Automates cage construction via render-and-reconstruct