ControlNeXt: Powerful and Efficient Control for Image and Video Generation

2024-08-12

arXiv.org

AI Summary

To address high computational overhead, low control precision, and poor training efficiency in controllable image/video generation, this paper proposes a lightweight and efficient novel architecture. Methodologically, it introduces three key innovations: (1) an ultra-minimalist plug-and-play design reducing parameter count by 90%; (2) the first Cross Normalization mechanism—replacing Zero-Convolution—to substantially improve training stability and convergence speed; and (3) native LoRA compatibility for flexible style-weight transfer. Evaluated across multiple diffusion-based image and video foundation models, the method achieves significant reductions in GPU memory consumption and FLOPs, surpasses mainstream approaches (e.g., ControlNet) in control accuracy, and accelerates training substantially. The architecture thus delivers both strong controllability and high deployment efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Diffusion models have demonstrated remarkable and robust abilities in both image and video generation. To achieve greater control over generated results, researchers introduce additional architectures, such as ControlNet, Adapters and ReferenceNet, to integrate conditioning controls. However, current controllable generation methods often require substantial additional computational resources, especially for video generation, and face challenges in training or exhibit weak control. In this paper, we propose ControlNeXt: a powerful and efficient method for controllable image and video generation. We first design a more straightforward and efficient architecture, replacing heavy additional branches with minimal additional cost compared to the base model. Such a concise structure also allows our method to seamlessly integrate with other LoRA weights, enabling style alteration without the need for additional training. As for training, we reduce up to 90% of learnable parameters compared to the alternatives. Furthermore, we propose another method called Cross Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and stable training convergence. We have conducted various experiments with different base models across images and videos, demonstrating the robustness of our method.

Problem

• Achieving efficient control in image and video generation.
• Reducing computational resources for controllable generation methods.
• Enhancing training stability and control strength in generative models.

Innovation

• Simplified architecture reduces computational costs
• Integrates with LoRA weights for style alteration
• Cross Normalization ensures stable training convergence