WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

2026-07-02

AI Summary

This work proposes a highly controllable video world model that decouples physical dynamics from pixel rendering by leveraging a large language model to coordinate 3D object trajectories and camera motion, which serve as disentangled semantic control signals for video generation. Unlike existing approaches that struggle to recover object identity after prolonged occlusion or support free-viewpoint manipulation, the proposed method enables persistent memory of precise visual identities even after objects remain off-screen for extended durations. It further supports unconstrained viewpoint exploration and controllable synthesis of complex events. Experimental results demonstrate that the resulting semantic-visual disentangled architecture generates complex scenes with strict physical coherence, consistent appearance, and long-term dynamic memory, significantly enhancing both controllability and realism in world modeling.

Abstract

We present WorldDirector, a highly controllable video world model framework designed for persistent dynamic object memory and unrestricted viewpoint exploration. Unlike existing world models that entangle physical dynamics with pixel rendering and rely on continuous visual observation to sustain motion, our framework explicitly decouples semantic motion orchestration from visual generation. By leveraging an LLM to coordinate 3D trajectories with camera movements and subsequently employing these orchestrated trajectories as control signals for video generation, our approach ensures strict physical logic and appearance stability, successfully preserving the exact visual identities of dynamic entities even when they re-enter the scene after prolonged periods out of view. Experimental results demonstrate that our method supports the synthesis of complex and extended events with unprecedented controllability and persistent dynamic object memory. Project Page: https://worlddirector.github.io/

Problem

• world models
• persistent memory
• dynamic objects
• controllable simulation
• viewpoint exploration

Innovation

• controllable world simulation
• persistent dynamic memory
• semantic motion orchestration
• decoupled visual generation
• LLM-guided trajectory planning