TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation

2024-10-08

arXiv.org

AI Summary

To address the challenge of high-fidelity fusion of multiple personalized concepts in diffusion models for image/video generation, this paper proposes a two-stage inference-time composition method. In the first stage, multi-objective perceptual sampling ensures the presence of each concept in the latent space; in the second stage, implicit appearance fusion is achieved in the denoised image space via the Tweedie formula. This work introduces the first fine-tuning-free, dynamic multi-concept fusion mechanism for diffusion models and pioneers the application of Tweedie distribution theory to concept synthesis therein. The framework naturally extends from text-to-image to image-to-video generation. Experiments demonstrate significant improvements in both conceptual fidelity and cross-concept consistency across multiple state-of-the-art diffusion models. Code and comprehensive results are publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Despite significant advancements in customizing text-to-image and video generation models, generating images and videos that effectively integrate multiple personalized concepts remains a challenging task. To address this, we present TweedieMix, a novel method for composing customized diffusion models during the inference phase. By analyzing the properties of reverse diffusion sampling, our approach divides the sampling process into two stages. During the initial steps, we apply a multiple object-aware sampling technique to ensure the inclusion of the desired target objects. In the later steps, we blend the appearances of the custom concepts in the de-noised image space using Tweedie's formula. Our results demonstrate that TweedieMix can generate multiple personalized concepts with higher fidelity than existing methods. Moreover, our framework can be effortlessly extended to image-to-video diffusion models, enabling the generation of videos that feature multiple personalized concepts. Results and source code are in our anonymous project page.

Problem

• Enhances multi-concept fusion in image/video generation
• Improves fidelity of personalized concept integration
• Extends framework to image-to-video diffusion models

Innovation

• Two-stage reverse diffusion sampling process
• Multiple object-aware sampling technique
• Tweedie's formula for concept blending