Perceive, Understand and Restore: Real-World Image Super-Resolution with Autoregressive Multimodal Generative Models

2025-03-14

AI Summary

Existing real-world image super-resolution (Real-ISR) methods suffer from reconstruction artifacts and structural distortions under complex or severe degradations, primarily due to diffusion models’ limited perceptual and structural understanding of low-quality inputs. To address this, we propose PURE, an end-to-end autoregressive multimodal reconstruction framework. PURE is the first to adapt pretrained autoregressive multimodal large models (e.g., Lumina-mGPT) to Real-ISR. It introduces an image-token entropy–based top-k sampling strategy to explicitly enforce local structural consistency, and integrates instruction tuning, cross-modal alignment, and semantic description–guided autoregressive token generation. Evaluated on multi-object complex scenes, PURE significantly improves content fidelity and texture realism, outperforming current state-of-the-art methods. The code and models are publicly released.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

By leveraging the generative priors from pre-trained text-to-image diffusion models, significant progress has been made in real-world image super-resolution (Real-ISR). However, these methods tend to generate inaccurate and unnatural reconstructions in complex and/or heavily degraded scenes, primarily due to their limited perception and understanding capability of the input low-quality image. To address these limitations, we propose, for the first time to our knowledge, to adapt the pre-trained autoregressive multimodal model such as Lumina-mGPT into a robust Real-ISR model, namely PURE, which Perceives and Understands the input low-quality image, then REstores its high-quality counterpart. Specifically, we implement instruction tuning on Lumina-mGPT to perceive the image degradation level and the relationships between previously generated image tokens and the next token, understand the image content by generating image semantic descriptions, and consequently restore the image by generating high-quality image tokens autoregressively with the collected information. In addition, we reveal that the image token entropy reflects the image structure and present a entropy-based Top-k sampling strategy to optimize the local structure of the image during inference. Experimental results demonstrate that PURE preserves image content while generating realistic details, especially in complex scenes with multiple objects, showcasing the potential of autoregressive multimodal generative models for robust Real-ISR. The model and code will be available at https://github.com/nonwhy/PURE.

Problem

• Enhances real-world image super-resolution accuracy
• Improves perception and understanding of degraded images
• Generates realistic details in complex multi-object scenes

Innovation

• Adapts Lumina-mGPT for image super-resolution
• Uses instruction tuning for image degradation perception
• Implements entropy-based Top-k sampling for structure optimization