DaBiT: Depth and Blur informed Transformer for Video Focal Deblurring

2024-07-01

AI Summary

This work introduces video refocusing—deblurring images to adjust focus depth—as a novel task addressing spatially continuous, depth-dependent in-focus/out-of-focus blur. Methodologically, we propose a depth- and blur-map-guided Transformer architecture featuring a blur-map-guided attention mechanism and a flow-based refocusing alignment module. To enable realistic training, we design a physically grounded synthetic pipeline that generates caustic-like defocus blur and construct DAVIS-Blur, the first benchmark dataset with pixel-level blur map annotations. Our model employs self-supervised pretraining and multi-scale reconstruction losses. Experiments demonstrate state-of-the-art performance on DAVIS-Blur, achieving a 1.9 dB PSNR improvement over prior methods. Both the code and dataset are publicly released.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In many real-world scenarios, recorded videos suffer from accidental focus blur, and while video deblurring methods exist, most specifically target motion blur or spatial-invariant blur. This paper introduces a framework optimized for the as yet unattempted task of video focal deblurring (refocusing). The proposed method employs novel map-guided transformers, in addition to image propagation, to effectively leverage the continuous spatial variance of focal blur and restore the footage. We also introduce a flow re-focusing module designed to efficiently align relevant features between blurry and sharp domains. Additionally, we propose a novel technique for generating synthetic focal blur data, broadening the model's learning capabilities and robustness to include a wider array of content. We have made a new benchmark dataset, DAVIS-Blur, available. This dataset, a modified extension of the popular DAVIS video segmentation set, provides realistic focal blur degradations as well as the corresponding blur maps. Comprehensive experiments demonstrate the superiority of our approach. We achieve state-of-the-art results with an average PSNR performance over 1.9dB greater than comparable existing video restoration methods. Our source code and the developed databases will be made available at https://github.com/crispianm/DaBiT

Problem

• Addresses video focal deblurring issues
• Enhances spatial variance in blur restoration
• Introduces synthetic data for model robustness

Innovation

• Map-guided transformers for deblurring
• Flow re-focusing module for alignment
• Synthetic focal blur data generation