Single Image Rolling Shutter Removal with Diffusion Models

2024-07-03

arXiv.org

AI Summary

This work addresses geometric distortion in single-frame rolling-shutter (RS) images caused by CMOS row-wise exposure. We propose the first diffusion-based single-frame RS correction method. Our core innovation is an “image-to-motion” mapping framework that directly regresses a physically consistent deformation field from a single RS image. To enhance local motion modeling, we design a patch-wise attention module. Furthermore, we introduce RS-Real—the first real-world RS dataset with synchronized IMU-calibrated ground-truth motion. The method learns an end-to-end mapping from RS images to optical flow/deformation fields without explicit motion priors. Evaluated on RS-Real, our approach significantly outperforms existing single-frame methods, achieving high-fidelity, physically interpretable distortion correction.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We present RS-Diffusion, the first Diffusion Models-based method for single-frame Rolling Shutter (RS) correction. RS artifacts compromise visual quality of frames due to the row wise exposure of CMOS sensors. Most previous methods have focused on multi-frame approaches, using temporal information from consecutive frames for the motion rectification. However, few approaches address the more challenging but important single frame RS correction. In this work, we present an ``image-to-motion'' framework via diffusion techniques, with a designed patch-attention module. In addition, we present the RS-Real dataset, comprised of captured RS frames alongside their corresponding Global Shutter (GS) ground-truth pairs. The GS frames are corrected from the RS ones, guided by the corresponding Inertial Measurement Unit (IMU) gyroscope data acquired during capture. Experiments show that our RS-Diffusion surpasses previous single RS correction methods. Our method and proposed RS-Real dataset lay a solid foundation for advancing the field of RS correction.

Problem

• Corrects rolling shutter artifacts in single images.
• Introduces diffusion models for image-to-motion transformation.
• Provides a new dataset for rolling shutter correction research.

Innovation

• Diffusion Models for single-frame RS correction
• Patch-attention module in image-to-motion framework
• RS-Real dataset with IMU-guided GS ground-truth