Couple to Control: Joint Initial Noise Design in Diffusion Models

2026-05-11

AI Summary

Diffusion models typically initialize with independent Gaussian noise, which constrains structural control and diversity in multi-sample generation. This work proposes explicitly modeling inter-sample dependencies by constructing coupled structures among initial noise variables while preserving standard Gaussian marginal distributions. The approach generalizes noise design from individual samples to joint distributions, unifying and extending existing strategies through novel mechanisms such as repulsive Gaussian coupling and subspace coupling. These techniques incur no additional sampling overhead and are compatible with Stable Diffusion architectures. Experiments demonstrate significant improvements in generation diversity across SD1.5, SDXL, and SD3, without compromising prompt alignment or image quality. The method also outperforms specialized inpainting approaches in background generation tasks and surpasses state-of-the-art noise optimization baselines under identical computational budgets.

Abstract

Diffusion models typically generate image batches from independent Gaussian initial noises. We argue that this independence assumption is only one choice within a broader class of valid joint noise designs. Instead, one can specify a coupling of the initial noises: each noise remains marginally standard Gaussian, so the pretrained diffusion model receives the same single-sample input distribution, while the dependence across samples is chosen by design. This reframes initial-noise control from selecting or optimizing individual seeds to designing the dependence structure of a multi-sample gallery. This view gives a general framework for initial-noise design, covering several existing methods as special cases and leading naturally to new coupled-noise constructions. Coupled noise can improve generation on its own without adding sampling cost, and it is flexible enough to serve as a structured initialization for optimization-based pipelines when additional computation is available. Empirically, repulsive Gaussian coupling improves gallery diversity on SD1.5, SDXL, and SD3 while largely preserving prompt alignment and image quality. It matches or outperforms recent test-time noise-optimization baselines on several diversity metrics at the same sampling cost as independent generation. Subspace couplings also support fixed-object background generation, producing diverse, natural backgrounds compared with specialized inpainting baselines, with a tunable trade-off in foreground fidelity.

Problem

• diffusion models
• initial noise
• coupling
• diversity
• generation control

Innovation

• coupled noise
• diffusion models
• initial noise design
• sample diversity
• structured initialization