From Open Loop to Closed Loop: A Test-Time Iterative Optimization Framework for Reference-Consistent Image Generation

2026-07-06

AI Summary

Existing controllable image generation methods predominantly operate as open-loop systems, lacking feedback mechanisms and thus struggling to rigorously ensure consistency between generated outputs and reference conditions. This work reframes the problem as a closed-loop dynamic tracking task and introduces, for the first time, the classical proportional-integral-derivative (PID) control algorithm from control theory. By iteratively optimizing latent control signals at inference time, the proposed approach enables real-time, training-free, and model-agnostic regulation of pretrained generative models. Built upon diffusion models, the method employs a sensor–controller architecture and demonstrates substantial improvements over open-loop baselines in tasks such as face identity preservation, pose control, and depth consistency—achieving up to a 25.36% gain in facial similarity and reducing pose and depth errors by 27.71% and 28.50%, respectively.

Abstract

While controllable image generation has made significant strides by incorporating visual reference conditions, existing methods predominantly operate as open-loop systems. They inject control signals in a strictly feed-forward manner, failing to guarantee strict fidelity to the reference due to the absence of active feedback and error correction mechanisms. To address this fundamental limitation, we propose a novel test-time iterative optimization framework that reformulates reference-consistent generation as a closed-loop dynamic tracking problem. By treating the pre-trained generative model as a control plant, our framework employs a sensor-controller architecture driven by a modified Proportional-Integral-Derivative (PID) algorithm. This mechanism iteratively optimizes the latent control signals at test time based on the sensed discrepancy between the generated output and the reference target. Notably, this approach is entirely training-free, model-agnostic, and integrates seamlessly around existing diffusion pipelines. Extensive evaluations across ID-preserving, pose-controlled, and depth-controlled generation tasks validate the universality of our method. Empirical results demonstrate improvements over computation-matched open-loop baselines, achieving relative performance gains of up to 25.36\% for facial similarity, alongside spatial error reductions of up to 27.71\% for pose alignment and 28.50\% for depth consistency. More broadly, this work offers a new conceptual perspective: it demonstrates that controllable generation can be effectively managed as a dynamic feedback system, bringing the rigorous principles of classical control theory into the optimization of generative models. Code is available at https://github.com/zzdrill/From-Open-Loop-to-Closed-Loop.

Problem

• reference-consistent image generation
• open-loop system
• feedback mechanism
• controllable generation
• fidelity to reference

Innovation

• closed-loop generation
• test-time optimization
• PID control
• reference-consistent image generation
• feedback mechanism