TD-Paint: Faster Diffusion Inpainting Through Time Aware Pixel Conditioning

2024-10-11

International Conference on Learning Representations

AI Summary

Diffusion-based image inpainting suffers from slow sampling and suboptimal conditional guidance, causing denoising trajectories to deviate from the underlying data manifold. To address this, we propose a time-aware pixel-level conditioning mechanism: it explicitly models the time-varying noise schedule during denoising and injects known pixel information early in the initial sampling stage—without modifying the network architecture or introducing auxiliary synchronization modules. This approach effectively steers the generative trajectory toward the true data manifold, achieving both accelerated sampling and high-fidelity reconstruction. Evaluated on three standard benchmarks, our method consistently outperforms state-of-the-art diffusion inpainting models, accelerating sampling by up to 2.3× while preserving reconstruction quality and reducing computational complexity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Diffusion models have emerged as highly effective techniques for inpainting, however, they remain constrained by slow sampling rates. While recent advances have enhanced generation quality, they have also increased sampling time, thereby limiting scalability in real-world applications. We investigate the generative sampling process of diffusion-based inpainting models and observe that these models make minimal use of the input condition during the initial sampling steps. As a result, the sampling trajectory deviates from the data manifold, requiring complex synchronization mechanisms to realign the generation process. To address this, we propose Time-aware Diffusion Paint (TD-Paint), a novel approach that adapts the diffusion process by modeling variable noise levels at the pixel level. This technique allows the model to efficiently use known pixel values from the start, guiding the generation process toward the target manifold. By embedding this information early in the diffusion process, TD-Paint significantly accelerates sampling without compromising image quality. Unlike conventional diffusion-based inpainting models, which require a dedicated architecture or an expensive generation loop, TD-Paint achieves faster sampling times without architectural modifications. Experimental results across three datasets show that TD-Paint outperforms state-of-the-art diffusion models while maintaining lower complexity.

Problem

• Slow sampling rates in diffusion-based inpainting models
• Inefficient use of input conditions during early sampling steps
• Complex synchronization mechanisms needed for trajectory realignment

Innovation

• Pixel-level variable noise modeling
• Early embedding of known pixels
• No architectural modifications needed