CineOrchestra: Unified Entity-Centric Conditioning for Cinematic Video Generation

2026-06-11

AI Summary

Existing text-to-video models struggle to jointly control multiple subjects, event timing, shot transitions, and camera motion. This work proposes a unified video diffusion model that, for the first time, represents heterogeneous cinematic elements—subjects, events, camera movements, and shots—as entity-centric conditional primitives operating over specific time intervals, while leveraging reference images to enhance visual consistency. The method introduces a parameter-free coordinated Rotary Position Embedding (RoPE) scheme that integrates interval-sampled temporal RoPE with 2D entity–time cross-attention RoPE, effectively addressing attention routing across both time and entities. Evaluated on two newly introduced benchmarks, the model significantly outperforms six specialized single-aspect baselines, achieving state-of-the-art performance in dense caption following, shot transition timing accuracy, and user preference studies.

Abstract

Cinematic video depicts multiple subjects acting or interacting at specific moments, captured with deliberate camera movement, and stitched together by shot transitions. Together, these elements demand a level of fine-grained control beyond current text-to-video models. Existing work addresses each axis in isolation: multi-subject personalization, temporal control, multi-shot synthesis, or camera control; no prior framework jointly integrates all four. We present CineOrchestra, a unified video diffusion model that controls subjects, events, cameras, and shot transitions simultaneously. Our key insight is that these heterogeneous cinematic elements share a fundamental structure: each is an entity acting over a specific temporal interval, which can therefore all be expressed through one shared structure of entity-centric conditioning primitives, augmented with reference images for visual entities. This formulation reduces the architectural challenge to a single positional encoding problem, which we solve with two parameter-free coordinated rotary embeddings: (a) an interval-sampled temporal RoPE that yields consistent attention behavior across events of dramatically varying duration, and (b) a 2D entity-temporal cross-attention RoPE that disambiguates per-entity conditions and routes each to its corresponding spatiotemporal region. On two new benchmarks, CineOrchestra outperforms six per-axis specialists on dense caption following and shot-transition timing, with consistent gains in a pairwise user study and component ablations.

Problem

• cinematic video generation
• multi-subject control
• temporal control
• camera control
• shot transitions

Innovation

• entity-centric conditioning
• unified video diffusion
• rotary position embedding
• multi-shot synthesis
• temporal control