World Narrative Model for Highly Controllable Video Generation: A Paradigm Shift from Pixel Sampling to Physical World Orchestration

2026-06-30

AI Summary

Existing video generation models lack quantitative controllability over geometric structure, motion dynamics, camera parameters, and lighting, limiting their utility in professional creative workflows. This work proposes the World Narrative Model (WNM), which pioneers a paradigm shift from pixel-level generation to physics-aware world orchestration by decoupling video synthesis into two stages: structured 4D physical narrative construction and neural rendering. Leveraging multi-agent collaboration, WNM integrates sparse multimodal inputs—such as text, sketches, and reference videos—to produce an editable, physically meaningful 4D world representation. This representation then drives either a frozen or lightly fine-tuned video foundation model acting as a neural shader. The approach substantially reduces reliance on random trial-and-error, enabling precise alignment with creator intent in terms of spatial layout, motion choreography, and cinematic language, thereby achieving high controllability, compatibility with professional pipelines, and strong scalability.

Abstract

The fundamental obstacle to industrial grade video generation is the lack of controllability: existing models treat video as a pixel distribution sampling problem, bypassing the explicit, instance level $4D$ $(3D + T)$ physical world. Consequently, content creators cannot specify geometry, motion, camera parameters, or lighting in a deterministic, quantitative way, leading to the infamous ''gacha'' loop that makes professional content creation prohibitively inefficient and expensive. To address this, we introduce the World Narrative Model (WNM), a paradigm that decouples what to render -- the structured physical narrative -- from how to render -- the pixel generation process. WNM replaces end-to-end black-box sampling with orchestrated $4D$ pre-visualization for media generation. Collaborative agents translate sparse multimodal inputs, including text, reference videos, and sketches, into a fully editable world representation with scene geometry, object layouts, character/animal skeleton motion, trajectories, camera motion, and lighting at quantitative, physically meaningful granularity. This representation acts as a deterministic structural blueprint that drives existing video foundation models, either frozen or lightly adapted, to render final footage, turning the base model into a faithful neural shader. Built on this engine, our human-AI platform supports automatic world generation and pre-visualization aligned with professional filmmaking pipelines, while director consoles enable seamless human refinement. Experiments show that WNM greatly reduces probabilistic ``gacha'' calls and produces videos whose layout, motion, and cinematography closely follow creator intent. The framework is open and modular, allowing each component, such as world representation, control agents, and adapters, to be independently improved. Project website: https://glassroom.sjtu.edu.cn/WNM/.

Problem

• controllability
• video generation
• 4D physical world
• pixel sampling
• creator intent

Innovation

• World Narrative Model
• 4D physical world orchestration
• controllable video generation
• structured world representation
• neural shader