Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control

2026-06-26

AI Summary

Existing autoregressive video generation methods often suffer from temporal degradation due to error accumulation over long-horizon inference, making it challenging to achieve high-fidelity, controllable synthesis of both human motion and camera trajectories simultaneously. This work proposes a fast autoregressive framework that decouples motion and camera control while sharing a unified video prior. By integrating slow-fast memory training, a t-guided dynamic projection mechanism, an enhanced Motion-CFG strategy, and a two-stage control architecture, the method enables, for the first time, precise compositional control over multiple human motions and camera paths. Trained on a large-scale multimodal annotated dataset, the approach significantly improves visual fidelity and control accuracy in long-horizon generation, outperforming current state-of-the-art methods across the board.

Abstract

Building interactive world models requires generating realistic videos while maintaining controllable dynamics over long horizons. Autoregressive video generation offers a scalable foundation, but suffers from error accumulation and temporal degradation during extended rollouts. This issue is further amplified under heterogeneous controls such as human motion and camera trajectories, which may interfere and destabilize a pretrained video prior, while existing methods often trade off controllability and visual quality. We propose "Directing the World", a fast autoregressive framework for controllable world-model video generation with compositional human-motion and camera-trajectory control. Our key idea is to decouple control learning while preserving a unified autoregressive video prior. We introduce a Fast-Slow Memory training strategy to stabilize long-horizon rollout learning and improve convergence. For human motion control, we design a t-guided Dynamic Projection mechanism and a refined Motion-CFG strategy, enabling temporally smooth and accurate motion alignment without degrading visual fidelity, and supporting multi-person control.After learning a robust motion prior, we introduce a second-stage camera-trajectory control module to compose human dynamics with viewpoint changes for coherent world exploration. We further construct a large-scale dataset with synchronized video, text, human-motion, and camera-trajectory annotations, organized into motion-centric and camera-centric subsets for decoupled training. Extensive experiments show stable long-horizon generation with precise controllability and high visual quality. See more at https://whydahuzi.github.io/Directing-the-World.github.io/.

Problem

• autoregressive video generation
• controllable dynamics
• human-motion control
• camera-trajectory control
• long-horizon video generation

Innovation

• autoregressive video generation
• compositional control
• human-motion control
• camera-trajectory control
• long-horizon video synthesis