Eliminating VAE for Fast and High-Resolution Generative Detail Restoration

2026-02-11

AI Summary

This work addresses the limitations of existing single-step diffusion-based super-resolution methods, which rely on variational autoencoders (VAEs), resulting in slow inference and high memory consumption that hinder high-resolution processing. For the first time, this study completely eliminates the VAE by reformulating the latent-space diffusion model into pixel space via pixel shuffling. To stabilize training, the authors introduce multi-stage adversarial distillation and a random padding strategy, along with a masked Fourier loss to mitigate abnormal magnitude artifacts. Combined with padding-based self-ensemble and classifier-free guidance, the proposed method achieves 4K super-resolution in under one second using only 6GB of GPU memory—2.8× faster and 60% less memory than GenDR—while preserving near-perfect visual quality and significantly outperforming current single-step diffusion super-resolution approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Diffusion models have attained remarkable breakthroughs in the real-world super-resolution (SR) task, albeit at slow inference and high demand on devices. To accelerate inference, recent works like GenDR adopt step distillation to minimize the step number to one. However, the memory boundary still restricts the maximum processing size, necessitating tile-by-tile restoration of high-resolution images. Through profiling the pipeline, we pinpoint that the variational auto-encoder (VAE) is the bottleneck of latency and memory. To completely solve the problem, we leverage pixel-(un)shuffle operations to eliminate the VAE, reversing the latent-based GenDR to pixel-space GenDR-Pix. However, upscale with x8 pixelshuffle may induce artifacts of repeated patterns. To alleviate the distortion, we propose a multi-stage adversarial distillation to progressively remove the encoder and decoder. Specifically, we utilize generative features from the previous stage models to guide adversarial discrimination. Moreover, we propose random padding to augment generative features and avoid discriminator collapse. We also introduce a masked Fourier space loss to penalize the outliers of amplitude. To improve inference performance, we empirically integrate a padding-based self-ensemble with classifier-free guidance to improve inference scaling. Experimental results show that GenDR-Pix performs 2.8x acceleration and 60% memory-saving compared to GenDR with negligible visual degradation, surpassing other one-step diffusion SR. Against all odds, GenDR-Pix can restore 4K image in only 1 second and 6GB.

Problem

• super-resolution
• diffusion models
• variational auto-encoder
• high-resolution image restoration
• inference acceleration

Innovation

• VAE elimination
• pixel-space diffusion
• multi-stage adversarial distillation
• random padding
• masked Fourier loss