MFSR-GAN: Multi-Frame Super-Resolution with Handheld Motion Modeling

2025-02-28

AI Summary

Smartphone handheld burst imaging suffers from low spatial resolution, severe noise, and complex motion due to sensor miniaturization; existing datasets fail to faithfully model the coupled degradation effects, hindering multi-frame super-resolution (MFSR) advancement. To address this, we propose: (1) the first synthetic data engine integrating multi-exposure static scenes with a physics-driven handheld motion model, accurately capturing realistic noise and non-rigid motion; (2) MFSR-GAN, a multi-scale RAW-to-RGB generation network centered on a reference frame, which explicitly decouples motion compensation from cross-domain reconstruction. Evaluated on both synthetic and real handheld bursts, our method significantly suppresses artifacts, yielding sharper and more natural reconstructions. It achieves state-of-the-art performance in PSNR, SSIM, and perceptual quality.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Smartphone cameras have become ubiquitous imaging tools, yet their small sensors and compact optics often limit spatial resolution and introduce distortions. Combining information from multiple low-resolution (LR) frames to produce a high-resolution (HR) image has been explored to overcome the inherent limitations of smartphone cameras. Despite the promise of multi-frame super-resolution (MFSR), current approaches are hindered by datasets that fail to capture the characteristic noise and motion patterns found in real-world handheld burst images. In this work, we address this gap by introducing a novel synthetic data engine that uses multi-exposure static images to synthesize LR-HR training pairs while preserving sensor-specific noise characteristics and image motion found during handheld burst photography. We also propose MFSR-GAN: a multi-scale RAW-to-RGB network for MFSR. Compared to prior approaches, MFSR-GAN emphasizes a"base frame"throughout its architecture to mitigate artifacts. Experimental results on both synthetic and real data demonstrates that MFSR-GAN trained with our synthetic engine yields sharper, more realistic reconstructions than existing methods for real-world MFSR.

Problem

• Overcoming smartphone camera resolution and distortion limitations
• Synthesizing realistic LR-HR training pairs with handheld motion
• Improving multi-frame super-resolution with a base frame approach

Innovation

• Synthetic data engine for LR-HR training pairs
• Multi-scale RAW-to-RGB network for MFSR
• Base frame emphasis to reduce artifacts