GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis

2023-12-18

arXiv.org

AI Summary

To address the challenge of balancing reconstruction speed and quality in real-time novel-view synthesis for monocular dynamic scenes, this paper proposes a forward-deformable learnable 3D Gaussian field. It employs template Gaussians as primitives, couples a time-aware forward deformation field to model non-rigid motion, and imposes a static prior to constrain deformation exclusively to moving regions. The method introduces the first forward-deformation Gaussian representation, designs an inductive-bias-aware initialization for explicit static-dynamic decoupling, and adopts end-to-end self-supervised optimization. Leveraging differentiable Gaussian rendering and self-supervised photometric losses, training on real-world scenes takes only ~20 minutes; real-time rendering achieves 96 FPS on an RTX 3090. Quantitatively, the approach outperforms state-of-the-art NeRF- and Gaussian-based methods in both PSNR and LPIPS metrics.

Abstract

We propose a method that achieves state-of-the-art rendering quality and efficiency on monocular dynamic scene reconstruction using deformable 3D Gaussians. Implicit deformable representations commonly model motion with a canonical space and time-dependent backward-warping deformation field. Our method, GauFRe, uses a forward-warping deformation to explicitly model non-rigid transformations of scene geometry. Specifically, we propose a template set of 3D Gaussians residing in a canonical space, and a time-dependent forward-warping deformation field to model dynamic objects. Additionally, we tailor a 3D Gaussian-specific static component supported by an inductive bias-aware initialization approach which allows the deformation field to focus on moving scene regions, improving the rendering of complex real-world motion. The differentiable pipeline is optimized end-to-end with a self-supervised rendering loss. Experiments show our method achieves competitive results and higher efficiency than both previous state-of-the-art NeRF and Gaussian-based methods. For real-world scenes, GauFRe can train in ~20 mins and offer 96 FPS real-time rendering on an RTX 3090 GPU. Project website: https://lynl7130.github.io/gaufre/index.html

Problem

• Dynamic Scene Reconstruction
• Complex Motion Handling
• Real-time Rendering

Innovation

• GauFRe
• real-time rendering
• dynamic scene reconstruction