Sample what you cant compress

2024-09-04

arXiv.org

AI Summary

Existing autoencoders often yield blurry reconstructions; while GANs or perceptual losses improve visual quality, they lack rigorous theoretical foundations. Diffusion models generate sharp images with strong theoretical grounding but remain largely disconnected from representation learning. This paper proposes SWYCC—a novel framework that enables the first end-to-end joint optimization of a continuous encoder-decoder architecture using diffusion-based loss. The encoder learns compact latent representations, while a stochastic decoder leverages diffusion priors to recover fine-grained, uncompressed details. Integrating variational inference with continuous latent space modeling, SWYCC achieves both high-fidelity reconstruction and strong generative capability. Experiments demonstrate that SWYCC surpasses GAN-based autoencoders in reconstruction quality, exhibits greater hyperparameter robustness, and yields latent representations inherently compatible with latent diffusion modeling—producing sharper, more photorealistic reconstructions.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

For learned image representations, basic autoencoders often produce blurry results. Reconstruction quality can be improved by incorporating additional penalties such as adversarial (GAN) and perceptual losses. Arguably, these approaches lack a principled interpretation. Concurrently, in generative settings diffusion has demonstrated a remarkable ability to create crisp, high quality results and has solid theoretical underpinnings (from variational inference to direct study as the Fisher Divergence). Our work combines autoencoder representation learning with diffusion and is, to our knowledge, the first to demonstrate the efficacy of jointly learning a continuous encoder and decoder under a diffusion-based loss. We demonstrate that this approach yields better reconstruction quality as compared to GAN-based autoencoders while being easier to tune. We also show that the resulting representation is easier to model with a latent diffusion model as compared to the representation obtained from a state-of-the-art GAN-based loss. Since our decoder is stochastic, it can generate details not encoded in the otherwise deterministic latent representation; we therefore name our approach"Sample what you can't compress", or SWYCC for short.

Problem

• Improving blurry image reconstruction in basic autoencoders
• Combining autoencoder representation learning with diffusion models
• Achieving higher compression and better generation quality

Innovation

• Combining autoencoder representation learning with diffusion
• Jointly learning continuous encoder and decoder
• Using stochastic decoder to generate missing details