Latent Bias Alignment for High-Fidelity Diffusion Inversion in Real-World Image Reconstruction and Manipulation

2026-03-24

AI Summary

Existing diffusion inversion methods suffer from limited reconstruction quality and poor robustness on real images due to misalignment between the inversion and generation trajectories, as well as incompatibility with the reconstruction process of vector-quantized autoencoders (VQAEs). To address these issues, this work proposes Latent-space Bias Optimization (LBO) and Image Latent-space Boosting (ILB), which introduce learnable latent-space biases at each inversion step to align the inversion trajectory with the generative one. This approach enables, for the first time, an approximate joint optimization of diffusion inversion and VQAE reconstruction. The proposed method significantly enhances reconstruction fidelity and demonstrates superior performance in downstream tasks such as image editing and rare concept generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent research has shown that text-to-image diffusion models are capable of generating high-quality images guided by text prompts. But can they be used to generate or approximate real-world images from the seed noise? This is known as the diffusion inversion problem, which serves as a fundamental building block for bridging diffusion models and real-world scenarios. However, existing diffusion inversion methods often suffer from low reconstruction quality or weak robustness. Two major challenges need to be carefully addressed: (1) the misalignment between the inversion and generation trajectories during the diffusion process, and (2) the mismatch between the diffusion inversion process and the VQ autoencoder (VQAE) reconstruction. To address these challenges, we introduce a latent bias vector at each inversion step, which is learned to reduce the misalignment between inversion and generation trajectories. We refer to this strategy as Latent Bias Optimization (LBO). Furthermore, we perform an approximate joint optimization of the diffusion inversion and VQAE reconstruction processes by learning to adjust the image latent representation, which serves as the connecting interface between them. We refer to this technique as Image Latent Boosting (ILB). Extensive experimental results demonstrate that the proposed method significantly improves the image reconstruction quality of the diffusion model, as well as the performance of downstream tasks, including image editing and rare concept generation.

Problem

• diffusion inversion
• real-world image reconstruction
• trajectory misalignment
• VQ autoencoder mismatch
• image manipulation

Innovation

• Latent Bias Optimization
• Image Latent Boosting
• Diffusion Inversion
• VQ Autoencoder
• Trajectory Alignment