Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo

2025-02-18

AI Summary

Existing energy-parameterized diffusion models suffer from architectural constraints and training instability, resulting in substantially inferior performance compared to score-matching or denoiser-based approaches. To address this, we propose an energy distillation paradigm: a pretrained diffusion model serves as the teacher to distill a stable, decoupled energy function; Helmholtz decomposition theoretically separates the energy field from the score field; and, for the first time, diffusion sampling is formulated as a Feynman–Kac sequential Monte Carlo process governed by an energy potential. Our framework enables low-temperature sampling, precise potential energy control, and compositional generation via multiple energy functions. Experiments demonstrate that our method achieves image synthesis and controllable generation quality and stability on par with state-of-the-art score-matching models, significantly enhancing both the practical utility and theoretical coherence of energy-based diffusion modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diffusion models may be formulated as a time-indexed sequence of energy-based models, where the score corresponds to the negative gradient of an energy function. As opposed to learning the score directly, an energy parameterization is attractive as the energy itself can be used to control generation via Monte Carlo samplers. Architectural constraints and training instability in energy parameterized models have so far yielded inferior performance compared to directly approximating the score or denoiser. We address these deficiencies by introducing a novel training regime for the energy function through distillation of pre-trained diffusion models, resembling a Helmholtz decomposition of the score vector field. We further showcase the synergies between energy and score by casting the diffusion sampling procedure as a Feynman Kac model where sampling is controlled using potentials from the learnt energy functions. The Feynman Kac model formalism enables composition and low temperature sampling through sequential Monte Carlo.

Problem

• Improving energy parameterized diffusion models
• Enhancing training stability via model distillation
• Enabling controlled sampling with Sequential Monte Carlo

Innovation

• Energy-based diffusion models
• Distillation training regime
• Sequential Monte Carlo sampling