Track the Noise, Move the World:3D-Grounded Motion-Consistent Noise for Controllable Video Generation

2026-07-02

AI Summary

Existing video generation models struggle to simultaneously and precisely control object motion and camera viewpoint changes within a single generation pass. This work proposes a novel approach that unifies the modeling of both factors in the diffusion model’s noise space. By constructing a 3D spatially aligned and motion-consistent noise representation, the method leverages sparse 3D point trajectories to deform reference-frame noise along target object trajectories and introduces spherical virtual noise to handle newly visible regions caused by viewpoint changes. For the first time, joint controllability of object trajectories and camera motion is achieved at the input noise level, without modifying the base model architecture or adding dedicated control modules. Merely through lightweight LoRA fine-tuning, the approach integrates with large foundation models such as Wan 2.1 (14B) to achieve state-of-the-art video quality and motion controllability on standard benchmarks.

Abstract

Modern image-and-text-to-video diffusion models can synthesize highly realistic videos by iteratively denoising an initial Gaussian noise tensor conditioned on reference image and text inputs. However, existing approaches still lack precise and unified controllability over both object motion and camera motion within a single generation process. We present UniCaMo, a unified framework that enables simultaneous control of object trajectories and camera viewpoints by directly constructing the input noise of the diffusion model. Specifically, UniCaMo builds a shared 3D-grounded motion-consistent noise space across latent video frames. Sparse 3D point tracks are used to warp the Gaussian noise of the reference frame along desired object trajectories, while a virtual spherical noise representation provides globally consistent noise values for newly revealed scene regions under camera motion. By combining local track-guided noise warping with global sphere-based noise sampling, UniCaMo maintains geometric and temporal consistency under both object movement and viewpoint changes. Because UniCaMo modifies only the input noise, it requires no auxiliary adapters, control branches, or architectural changes to the underlying video diffusion model. With lightweight LoRA fine-tuning on large pretrained video diffusion models, including Wan 2.1 (14B), UniCaMo achieves state-of-the-art results in both video quality and motion controllability on standard controllable video generation benchmarks.

Problem

• controllable video generation
• object motion
• camera motion
• motion consistency
• 3D grounding

Innovation

• 3D-grounded noise
• motion-consistent noise
• controllable video generation
• camera motion control
• object trajectory control