SILO: Solving Inverse Problems with Latent Operators

2025-01-20

AI Summary

Diffusion models for image inverse problems suffer from high computational overhead and limited reconstruction quality due to repeated autoencoder invocations. To address this, we propose an implicit degradation modeling framework in latent space: the degradation process is explicitly shifted from pixel space to latent space, and a lightweight, end-to-end learnable degradation operator is jointly optimized. This design restricts autoencoder usage to only the initial and final sampling steps, drastically reducing encoding/decoding overhead. Our method integrates latent diffusion models, implicit degradation operators, and a lightweight autoencoder within a unified optimization pipeline. Extensive experiments on deblurring, super-resolution, and denoising demonstrate consistent superiority over state-of-the-art methods, achieving higher PSNR and SSIM scores while accelerating single-sample inference by up to 2.1×.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Consistent improvement of image priors over the years has led to the development of better inverse problem solvers. Diffusion models are the newcomers to this arena, posing the strongest known prior to date. Recently, such models operating in a latent space have become increasingly predominant due to their efficiency. In recent works, these models have been applied to solve inverse problems. Working in the latent space typically requires multiple applications of an Autoencoder during the restoration process, which leads to both computational and restoration quality challenges. In this work, we propose a new approach for handling inverse problems with latent diffusion models, where a learned degradation function operates within the latent space, emulating a known image space degradation. Usage of the learned operator reduces the dependency on the Autoencoder to only the initial and final steps of the restoration process, facilitating faster sampling and superior restoration quality. We demonstrate the effectiveness of our method on a variety of image restoration tasks and datasets, achieving significant improvements over prior art.

Problem

• Diffusion Models
• Autoencoders
• Image Steganography

Innovation

• Diffusion Models
• Hidden Information Processing
• Performance Enhancement