QMambaBSR: Burst Image Super-Resolution with Query State Space Model

2024-08-16

arXiv.org

AI Summary

This work addresses three key challenges in burst super-resolution: (1) vulnerability of sub-pixel features to noise, (2) insufficient modeling of inter-frame and intra-frame correlations, and (3) detail blurring and artifacts induced by fixed upscaling. To this end, we propose a novel multi-frame joint reconstruction framework. Our contributions are threefold: (1) the Query-based State Space Model (QSSM), enabling single-step, noise-robust sub-pixel feature extraction; (2) the Adaptive Upsampling module (AdaUp), which dynamically perceives multi-frame sub-pixel spatial distributions and optimizes fusion weights; and (3) a joint modeling mechanism integrating cross-frame queries and intra-frame scanning, enhanced by dynamic deformable convolutions for improved local representation. Evaluated on four mainstream synthetic and real-world datasets, our method achieves state-of-the-art performance, significantly enhancing high-frequency detail recovery while effectively suppressing over-smoothing and reconstruction artifacts.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Burst super-resolution aims to reconstruct high-resolution images with higher quality and richer details by fusing the sub-pixel information from multiple burst low-resolution frames. In BusrtSR, the key challenge lies in extracting the base frame's content complementary sub-pixel details while simultaneously suppressing high-frequency noise disturbance. Existing methods attempt to extract sub-pixels by modeling inter-frame relationships frame by frame while overlooking the mutual correlations among multi-current frames and neglecting the intra-frame interactions, leading to inaccurate and noisy sub-pixels for base frame super-resolution. Further, existing methods mainly employ static upsampling with fixed parameters to improve spatial resolution for all scenes, failing to perceive the sub-pixel distribution difference across multiple frames and cannot balance the fusion weights of different frames, resulting in over-smoothed details and artifacts. To address these limitations, we introduce a novel Query Mamba Burst Super-Resolution (QMambaBSR) network, which incorporates a Query State Space Model (QSSM) and Adaptive Up-sampling module (AdaUp). Specifically, based on the observation that sub-pixels have consistent spatial distribution while random noise is inconsistently distributed, a novel QSSM is proposed to efficiently extract sub-pixels through inter-frame querying and intra-frame scanning while mitigating noise interference in a single step. Moreover, AdaUp is designed to dynamically adjust the upsampling kernel based on the spatial distribution of multi-frame sub-pixel information in the different burst scenes, thereby facilitating the reconstruction of the spatial arrangement of high-resolution details. Extensive experiments on four popular synthetic and real-world benchmarks demonstrate that our method achieves a new state-of-the-art performance.

Problem

• Extract sub-pixel details while suppressing noise in burst super-resolution.
• Model inter-frame and intra-frame interactions for accurate sub-pixel extraction.
• Dynamically adjust upsampling to balance fusion weights and reduce artifacts.

Innovation

• Query State Space Model for sub-pixel extraction
• Adaptive Up-sampling for dynamic resolution adjustment
• Inter-frame querying and intra-frame scanning integration