SONIC: Spectral Optimization of Noise for Inpainting with Consistency

2025-11-25

AI Summary

This work addresses zero-shot image inpainting—restoring masked regions without model training. We propose a novel approach leveraging off-the-shelf text-to-image diffusion models, wherein the initial noise is spectrally optimized to align with the content distribution of unmasked regions, while linearized latent-space guidance accelerates the diffusion process. Crucially, no model fine-tuning or costly backward unfolding is required; high-fidelity restoration is achieved within tens of sampling steps. Our key contribution is the first integration of spectral-domain noise initialization with training-free diffusion guidance, enabling superior detail fidelity and seamless boundary coherence while preserving semantic consistency. Extensive experiments demonstrate that our method outperforms existing zero-shot and lightweight fine-tuning approaches across diverse inpainting tasks, achieving state-of-the-art visual quality and content consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting, in practice, their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue that the missing ingredient for training-free inpainting is the optimization (guidance) of the initial seed noise. We propose to optimize the initial seed noise to approximately match the unmasked parts of the data - with as few as a few tens of optimization steps. We then apply conventional training-free inpainting methods on top of our optimized initial seed noise. Critically, we propose two core ideas to effectively implement this idea: (i) to avoid the costly unrolling required to relate the initial noise and the generated outcome, we perform linear approximation; and (ii) to stabilize the optimization, we optimize the initial seed noise in the spectral domain. We demonstrate the effectiveness of our method on various inpainting tasks, outperforming the state of the art. Project page: https://ubc-vision.github.io/sonic/

Problem

• Optimizes initial noise for training-free image inpainting
• Uses spectral domain and linear approximation for efficiency
• Enhances off-the-shelf models without specialized inpainting training

Innovation

• Optimizes initial seed noise for inpainting
• Uses linear approximation to avoid unrolling
• Performs noise optimization in spectral domain