UniCon: Unidirectional Information Flow for Effective Control of Large-Scale Diffusion Models

2025-03-21

AI Summary

To address the low training efficiency and high GPU memory consumption of control adapters in large-scale diffusion models, this paper proposes a unidirectional information flow architecture: the control adapter performs forward computation independently to generate outputs, while the backbone diffusion model is excluded from gradient computation—achieving full decoupling of training. This novel unidirectional control flow mechanism overcomes ControlNet’s capacity bottleneck, enabling a twofold increase in controller parameter count under identical GPU resources. Integrated with gradient flow redirection and memory-optimized training, our approach reduces GPU memory usage by 33% and accelerates training by 2.3×. It achieves state-of-the-art control accuracy and image generation quality, demonstrating superior performance on multi-condition image synthesis tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce UniCon, a novel architecture designed to enhance control and efficiency in training adapters for large-scale diffusion models. Unlike existing methods that rely on bidirectional interaction between the diffusion model and control adapter, UniCon implements a unidirectional flow from the diffusion network to the adapter, allowing the adapter alone to generate the final output. UniCon reduces computational demands by eliminating the need for the diffusion model to compute and store gradients during adapter training. Our results indicate that UniCon reduces GPU memory usage by one-third and increases training speed by 2.3 times, while maintaining the same adapter parameter size. Additionally, without requiring extra computational resources, UniCon enables the training of adapters with double the parameter volume of existing ControlNets. In a series of image conditional generation tasks, UniCon has demonstrated precise responsiveness to control inputs and exceptional generation capabilities.

Problem

• Enhance control and efficiency in large-scale diffusion models
• Reduce computational demands during adapter training
• Enable training of larger adapters without extra resources

Innovation

• Unidirectional flow from diffusion network to adapter
• Reduces GPU memory usage by one-third
• Increases training speed by 2.3 times