Gyro-based Neural Single Image Deblurring

2024-04-01

arXiv.org

AI Summary

This work addresses the ill-posed problem of single-image motion deblurring. We propose GyroDeblurNet—the first deep deblurring framework to tightly integrate raw gyroscope measurements. Methodologically, we design a Gyroscope Refinement Block and a Gyroscope-guided Deblurring Block to jointly model camera motion and image blur; we further introduce a novel motion embedding representation and a curriculum learning strategy, enabling the first robust utilization of noisy real-world gyroscope signals. Extensive experiments on both synthetic and real-world datasets demonstrate that our approach significantly improves restoration accuracy and structural fidelity, achieving state-of-the-art (SOTA) performance. By effectively leveraging inertial sensor data, GyroDeblurNet establishes a new paradigm for sensor-assisted image restoration.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper, we present GyroDeblurNet, a novel single-image deblurring method that utilizes a gyro sensor to resolve the ill-posedness of image deblurring. The gyro sensor provides valuable information about camera motion that can improve deblurring quality. However, exploiting real-world gyro data is challenging due to errors from various sources. To handle these errors, GyroDeblurNet is equipped with two novel neural network blocks: a gyro refinement block and a gyro deblurring block. The gyro refinement block refines the erroneous gyro data using the blur information from the input image. The gyro deblurring block removes blur from the input image using the refined gyro data and further compensates for gyro error by leveraging the blur information from the input image. For training a neural network with erroneous gyro data, we propose a training strategy based on the curriculum learning. We also introduce a novel gyro data embedding scheme to represent real-world intricate camera shakes. Finally, we present both synthetic and real-world datasets for training and evaluating gyro-based single image deblurring. Our experiments demonstrate that our approach achieves state-of-the-art deblurring quality by effectively utilizing erroneous gyro data.

Problem

• Resolves ill-posed image deblurring using gyro sensor data
• Handles gyro sensor errors via neural refinement and deblurring blocks
• Improves deblurring quality by embedding real-world camera shake data

Innovation

• Uses gyro sensor for camera motion data
• Refines gyro data with neural network blocks
• Employs curriculum learning for training