SkyReels-A2: Compose Anything in Video Diffusion Transformers

2025-04-03

AI Summary

This paper introduces Element-to-Video (E2V), a novel task paradigm for controllable video generation conditioned on text prompts and multiple reference images—ensuring strict visual fidelity of characters, objects, and backgrounds while preserving spatiotemporal coherence and natural motion. Methodologically, we propose a joint image-text embedding model to simultaneously optimize element-level fidelity and global semantic consistency. We establish A2 Bench, the first dedicated E2V benchmark, and open-source the first production-grade E2V model, integrating a video diffusion Transformer, a multi-reference image conditioning mechanism, and a prompt–reference–video triplet data pipeline, with optimized inference scheduling. Experiments demonstrate that our approach significantly outperforms leading closed-source commercial models on A2 Bench. The framework has been successfully deployed in real-world applications, including theatrical production and virtual e-commerce.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper presents SkyReels-A2, a controllable video generation framework capable of assembling arbitrary visual elements (e.g., characters, objects, backgrounds) into synthesized videos based on textual prompts while maintaining strict consistency with reference images for each element. We term this task elements-to-video (E2V), whose primary challenges lie in preserving the fidelity of each reference element, ensuring coherent composition of the scene, and achieving natural outputs. To address these, we first design a comprehensive data pipeline to construct prompt-reference-video triplets for model training. Next, we propose a novel image-text joint embedding model to inject multi-element representations into the generative process, balancing element-specific consistency with global coherence and text alignment. We also optimize the inference pipeline for both speed and output stability. Moreover, we introduce a carefully curated benchmark for systematic evaluation, i.e, A2 Bench. Experiments demonstrate that our framework can generate diverse, high-quality videos with precise element control. SkyReels-A2 is the first open-source commercial grade model for the generation of E2V, performing favorably against advanced closed-source commercial models. We anticipate SkyReels-A2 will advance creative applications such as drama and virtual e-commerce, pushing the boundaries of controllable video generation.

Problem

• Generate videos from arbitrary visual elements and text prompts
• Maintain strict consistency with reference images for each element
• Balance element-specific consistency with global coherence and text alignment

Innovation

• Image-text joint embedding for multi-element control
• Optimized inference pipeline for speed and stability
• Comprehensive data pipeline for training triplets