InpaintSLat: Inpainting Structured 3D Latents via Initial Noise Optimization

2026-05-01

AI Summary

Existing structured 3D latent diffusion models struggle with inpainting tasks due to the high sensitivity of initial noise to geometry, often failing to ensure strict alignment with surrounding context. This work proposes a training-free 3D controllable inpainting method that, for the first time, treats initial noise optimization as a control dimension independent of the sampling trajectory. By combining a backward propagation approximation derived from rectified flow models with a spectral parameterization strategy specifically designed for 3D latent variables, the method efficiently optimizes the initial noise. It achieves high-fidelity completion while significantly outperforming existing training-free baselines, demonstrating marked improvements in contextual consistency and alignment with text prompts.

Abstract

We present a training-free approach for controllable 3D inpainting based on initial noise optimization. In the structured 3D latent diffusion framework, we observe that the underlying geometric structure is established during the early stages of the diffusion process and exhibits high sensitivity to the initial noise. Such characteristics compromise stability in tasks like inpainting and editing, where the model must ensure strict alignment with the existing context while synthesizing a new structure. In this paper, we introduce a strategy to optimize the initial noise within the structured 3D latent diffusion framework, ensuring high-fidelity 3D inpainting. Specifically, we update the initial noise by leveraging a backpropagation approximation grounded in the rectified flow model, with the spectral parameterization specially designed for robust and efficient structured 3D latent optimization. Experiments demonstrate consistent improvements in contextual consistency and prompt alignment over representative training-free inpainting baselines, establishing initial noise control as an independent dimension for 3D inpainting, orthogonal to conventional sampling trajectory manipulation.

Problem

• 3D inpainting
• structured 3D latent
• initial noise
• contextual consistency
• diffusion model

Innovation

• initial noise optimization
• structured 3D latent
• training-free inpainting
• rectified flow
• spectral parameterization