Deblur4DGS: 4D Gaussian Splatting from Blurry Monocular Video

2024-12-09

arXiv.org

AI Summary

Existing 4D reconstruction methods for blurry monocular videos struggle to accurately model continuous dynamics during exposure due to coupled camera jitter and object motion, leading to motion blur. This work introduces the first 4D extension of 3D Gaussian Splatting—termed 4D Gaussian Splatting—designed specifically to address motion blur in dynamic scenes. Instead of conventional continuous motion representations, it models scene elements via explicit trajectory parameterization; replaces explicit motion estimation with exposure-time-aware modeling; employs blur-aware, variable-canonical Gaussians; and incorporates exposure regularization alongside multi-frame and multi-resolution consistency constraints. Evaluated on four downstream tasks—novel view synthesis, video deblurring, frame interpolation, and video stabilization—the framework consistently outperforms state-of-the-art 4D reconstruction approaches, enabling high-fidelity dynamic scene reconstruction and effective enhancement of multi-view blurry video sequences.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent 4D reconstruction methods have yielded impressive results but rely on sharp videos as supervision. However, motion blur often occurs in videos due to camera shake and object movement, while existing methods render blurry results when using such videos for reconstructing 4D models. Although a few NeRF-based approaches attempted to address the problem, they struggled to produce high-quality results, due to the inaccuracy in estimating continuous dynamic representations within the exposure time. Encouraged by recent works in 3D motion trajectory modeling using 3D Gaussian Splatting (3DGS), we suggest taking 3DGS as the scene representation manner, and propose the first 4D Gaussian Splatting framework to reconstruct a high-quality 4D model from blurry monocular video, named Deblur4DGS. Specifically, we transform continuous dynamic representations estimation within an exposure time into the exposure time estimation. Moreover, we introduce exposure regularization to avoid trivial solutions, as well as multi-frame and multi-resolution consistency ones to alleviate artifacts. Furthermore, to better represent objects with large motion, we suggest blur-aware variable canonical Gaussians. Beyond novel-view synthesis, Deblur4DGS can be applied to improve blurry video from multiple perspectives, including deblurring, frame interpolation, and video stabilization. Extensive experiments on the above four tasks show that Deblur4DGS outperforms state-of-the-art 4D reconstruction methods. The codes are available at https://github.com/ZcsrenlongZ/Deblur4DGS.

Problem

• Reconstructs 4D models from blurry monocular videos
• Estimates exposure time for dynamic scene representation
• Improves blurry video via deblurring and stabilization

Innovation

• Uses 3D Gaussian Splatting for 4D reconstruction
• Transforms dynamic estimation into exposure estimation
• Introduces blur-aware variable canonical Gaussians