LooseControlVideo: Directorial Video Control using Spatial Blocking

2026-06-17

AI Summary

This work addresses the challenges in text-to-video generation involving multi-object scenes—namely, the difficulty of 3D spatial arrangement, the tight coupling between semantic layout and dynamics, and the reliance of existing methods on dense frame-level guidance—by proposing a sparse, oriented 3D bounding box-based video control framework. Treating 3D boxes as “scheduling” proxies, the framework enables users to intuitively specify high-level object layouts and trajectories, while the model automatically synthesizes realistic occlusions, motion, and interactions. The approach introduces a novel DNOCS encoding to jointly model object scale, orientation, and depth-ordered occlusion relationships, and is fine-tuned on the Wan 2.2 backbone. Evaluated on nuScenes, HO-3D, and BEHAVE benchmarks, it significantly outperforms 2D bounding box and optical flow baselines, reducing trajectory error by 1.2–3×, improving rigid motion consistency by 2×, and increasing occlusion accuracy by 1.5–2×.

Abstract

Precise 3D spatial orchestration in text-to-video generation remains a significant challenge, particularly for multi-object scenes where semantic layout and temporal dynamics are often entangled. While existing depth-conditioned models achieve good structural fidelity, they necessitate dense, frame-accurate guidance that is labor-intensive to author for dynamic events involving deformable objects. We present LooseControlVideo, a framework that enables intuitive and expressive control by using sparse, oriented 3D boxes as a "blocking" proxy. This allows users to author high-level layout and trajectory while leveraging a video generative model to generate realistic occlusions, dynamics and interactions. We achieve this by fine-tuning a Wan 2.2 backbone on a video dataset annotated with DNOCS, a novel encoding for 3D size, orientation and depth-ordered occlusions. Furthermore, our method allows for localized refinement, such as adjusting a jump trajectory or adding an interaction, with minimal disruption to the global scene context. Extensive evaluations on the nuScenes, HO-3D, and BEHAVE benchmarks demonstrate that LooseControlVideo significantly outperforms existing 2D-box and flow-based baselines. Our findings indicate a 1.2x to 3x improvement in Trajectory Error; 2x improvement in Rigid Motion Consistency; and a 1.5x to 2x increase in Occlusion Accuracy over current state-of-the-art layout-conditioned models, demonstrating that oriented 3D primitives provide good geometric prior for complex, multi-agent video authoring.

Problem

• text-to-video generation
• 3D spatial orchestration
• multi-object scenes
• dynamic events
• deformable objects

Innovation

• oriented 3D boxes
• spatial blocking
• DNOCS encoding
• video generation control
• occlusion-aware synthesis