LISA: Likelihood Score Alignment for Visual-condition Controllable Generation

2026-06-25

AI Summary

Existing dual-branch generative models lack theoretical understanding of the side network’s role and training efficiency, limiting their performance in conditional controlled generation. This work addresses this gap by interpreting the main network as an unconditional score prior and the side network as implicitly modeling the likelihood score within the framework of score-based generative models. We propose LISA, a novel method that employs a lightweight decoder to project intermediate features from the side network into score space, thereby constructing an approximate likelihood score objective. LISA jointly optimizes diffusion loss and an alignment regularization loss, introducing negligible training overhead and no additional inference cost. Extensive experiments demonstrate that LISA significantly accelerates convergence, improves generation quality, and enhances conditional disentanglement of side network features across diverse image and video tasks, architectures, and both diffusion and flow-matching models.

Abstract

The prevalent dual-branch paradigm, i.e., training a side network to encode visual conditions and fusing its intermediate-layer features to a frozen pretrained main network, has shown remarkable success in visual-condition controllable generation. Despite its widespread adoption, the role of the side branch and its training efficiency remain underexplored. In this paper, we first revisit this mainstream paradigm through the lens of score-based generative modeling: 1) The main network preserves visual perceptual quality by providing a prior unconditional score. 2) The side network steers conditional control by implicitly contributing a likelihood score. Guided by this perspective, we propose LIkelihood Score Alignment (LISA), an effective regularization method that explicitly aligns the intermediate feature of the side network with an approximated likelihood score. Specifically, we first hook features from a designated layer of the side network and project them into the score latent space by a lightweight decoder. Then, we construct an approximated likelihood score target and calculate the distance between the decoder's output and this target as an additional regularization loss. Finally, we jointly optimize the side network and decoder with both standard diffusion loss and our regularization loss. Experiments across various image/video tasks, architectures, and diffusion/flow models demonstrated that LISA can not only consistently accelerate the training convergence and improve final synthetic results, but also encourage the side network's features to be more disentangled for conditional modeling with negligible additional training cost and zero extra inference cost.

Problem

• visual-condition controllable generation
• dual-branch paradigm
• side network
• training efficiency
• score-based generative modeling

Innovation

• score-based generative modeling
• likelihood score alignment
• visual-condition controllable generation
• dual-branch paradigm
• feature disentanglement