DyBluRF: Dynamic Deblurring Neural Radiance Fields for Blurry Monocular Video

2023-12-21

arXiv.org

AI Summary

To address spatiotemporal detail degradation caused by motion blur in novel view synthesis from blurry monocular videos, this paper proposes a two-stage dynamic deblurring NeRF framework. First, coarse scene reconstruction is achieved via Basic Ray Initialization (BRI). Second, global camera motion and local object motion are decoupled, enabling incremental prediction of sharp rays in latent space (ILSP) without mask supervision—effectively separating static and dynamic components. The method incorporates geometric regularization and motion-decoupling losses to enhance reconstruction consistency. Evaluated on multiple blurry monocular video datasets, our approach significantly outperforms state-of-the-art methods, achieving superior performance across all major metrics: PSNR, SSIM, and LPIPS. It generates high-fidelity, spatiotemporally consistent, and artifact-free novel views.

Abstract

Neural Radiance Fields (NeRF), initially developed for static scenes, have inspired many video novel view synthesis techniques. However, the challenge for video view synthesis arises from motion blur, a consequence of object or camera movement during exposure, which hinders the precise synthesis of sharp spatio-temporal views. In response, we propose a novel dynamic deblurring NeRF framework for blurry monocular video, called DyBluRF, consisting of a Base Ray Initialization (BRI) stage and a Motion Decomposition-based Deblurring (MDD) stage. Our DyBluRF is the first that handles the novel view synthesis for blurry monocular video with a novel two-stage framework. In the BRI stage, we coarsely reconstruct dynamic 3D scenes and jointly initialize the base ray, which is further used to predict latent sharp rays, using the inaccurate camera pose information from the given blurry frames. In the MDD stage, we introduce a novel Incremental Latent Sharp-rays Prediction (ILSP) approach for the blurry monocular video frames by decomposing the latent sharp rays into global camera motion and local object motion components. We further propose two loss functions for effective geometry regularization and decomposition of static and dynamic scene components without any mask supervision. Experiments show that DyBluRF outperforms qualitatively and quantitatively the SOTA methods.

Problem

• Addresses motion blur in video view synthesis
• Deblurs monocular video using neural radiance fields
• Decomposes global and local motion for sharp reconstruction

Innovation

• Motion Deblurring NeRF for blurry video
• Base Ray Initialization and Motion Decomposition
• Incremental Latent Sharp-rays Prediction approach