VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation

2025-02-11

AI Summary

Existing image-to-video generation methods only support coarse-grained control over one or two visual elements (e.g., camera or object motion), failing to jointly and precisely regulate multiple critical factors—such as camera motion, object motion, and lighting direction. To address this limitation, we propose the first framework enabling fine-grained, coordinated control over all three elements. Our approach introduces a spatial triple-attention Transformer architecture, constructs VLD—the first synthetic video dataset with explicit lighting-direction annotations—and proposes a novel three-stage decoupled training paradigm that eliminates the need for joint multi-attribute annotation. By integrating multimodal conditioning (text, image, and lighting), our method enables physically plausible modeling of light transmission and reflection. Extensive experiments demonstrate significant improvements over state-of-the-art methods across multiple benchmarks, achieving superior control accuracy, temporal consistency, visual fidelity, and physical plausibility.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent image-to-video generation methods have demonstrated success in enabling control over one or two visual elements, such as camera trajectory or object motion. However, these methods are unable to offer control over multiple visual elements due to limitations in data and network efficacy. In this paper, we introduce VidCRAFT3, a novel framework for precise image-to-video generation that enables control over camera motion, object motion, and lighting direction simultaneously. To better decouple control over each visual element, we propose the Spatial Triple-Attention Transformer, which integrates lighting direction, text, and image in a symmetric way. Since most real-world video datasets lack lighting annotations, we construct a high-quality synthetic video dataset, the VideoLightingDirection (VLD) dataset. This dataset includes lighting direction annotations and objects of diverse appearance, enabling VidCRAFT3 to effectively handle strong light transmission and reflection effects. Additionally, we propose a three-stage training strategy that eliminates the need for training data annotated with multiple visual elements (camera motion, object motion, and lighting direction) simultaneously. Extensive experiments on benchmark datasets demonstrate the efficacy of VidCRAFT3 in producing high-quality video content, surpassing existing state-of-the-art methods in terms of control granularity and visual coherence. All code and data will be publicly available. Project page: https://sixiaozheng.github.io/VidCRAFT3/.

Problem

• Control multiple visual elements in video generation.
• Precise control over camera, object, and lighting.
• Generate high-quality videos without multi-element annotations.

Innovation

• Simultaneous control over multiple visual elements
• Spatial Triple-Attention Transformer integration
• High-quality synthetic video dataset creation