CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation

2025-02-12

AI Summary

This work addresses the weak 3D awareness and limited spatial/camera controllability in text-to-video generation, proposing the first 3D-controllable video generation framework tailored for cinematic production. Methodologically: (1) it introduces a two-stage interactive 3D conditioning pipeline enabling precise object placement, parametric camera trajectory specification, and compositional control in 3D space; (2) it establishes the first large-scale automated 3D video annotation pipeline, jointly modeling depth, camera pose, and semantic segmentation for multi-attribute controllability; (3) it integrates multi-view depth-aware rendering with diffusion-based conditional guidance to achieve 3D-aware video distillation. Experiments demonstrate state-of-the-art performance in 3D consistency, motion controllability, and text-video alignment, significantly advancing professional-grade cinematic scene layout generation.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this work, we present CineMaster, a novel framework for 3D-aware and controllable text-to-video generation. Our goal is to empower users with comparable controllability as professional film directors: precise placement of objects within the scene, flexible manipulation of both objects and camera in 3D space, and intuitive layout control over the rendered frames. To achieve this, CineMaster operates in two stages. In the first stage, we design an interactive workflow that allows users to intuitively construct 3D-aware conditional signals by positioning object bounding boxes and defining camera movements within the 3D space. In the second stage, these control signals--comprising rendered depth maps, camera trajectories and object class labels--serve as the guidance for a text-to-video diffusion model, ensuring to generate the user-intended video content. Furthermore, to overcome the scarcity of in-the-wild datasets with 3D object motion and camera pose annotations, we carefully establish an automated data annotation pipeline that extracts 3D bounding boxes and camera trajectories from large-scale video data. Extensive qualitative and quantitative experiments demonstrate that CineMaster significantly outperforms existing methods and implements prominent 3D-aware text-to-video generation. Project page: https://cinemaster-dev.github.io/.

Problem

• 3D-aware text-to-video generation
• User-controlled object and camera manipulation
• Automated 3D data annotation from videos

Innovation

• 3D-aware text-to-video generation
• Interactive 3D object and camera control
• Automated 3D data annotation pipeline