DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data

2026-04-02

AI Summary

Existing video diffusion models struggle to generate realistic and controllable videos due to the scarcity of training samples exhibiting highly dynamic or precisely controlled motion. To address this limitation, this work proposes DynaVid, a two-stage diffusion framework that decouples motion from appearance. The first stage leverages synthetic optical flow—generated via computer graphics—as a supervisory signal to learn controllable dynamics; the second stage conditions on this learned motion representation to synthesize high-fidelity video frames. By explicitly separating motion modeling from appearance generation, DynaVid significantly enhances both visual realism and motion controllability, particularly in challenging scenarios involving vigorous human actions or extreme camera movements, outperforming current state-of-the-art methods.

Abstract

Despite recent progress, video diffusion models still struggle to synthesize realistic videos involving highly dynamic motions or requiring fine-grained motion controllability. A central limitation lies in the scarcity of such examples in commonly used training datasets. To address this, we introduce DynaVid, a video synthesis framework that leverages synthetic motion data in training, which is represented as optical flow and rendered using computer graphics pipelines. This approach offers two key advantages. First, synthetic motion offers diverse motion patterns and precise control signals that are difficult to obtain from real data. Second, unlike rendered videos with artificial appearances, rendered optical flow encodes only motion and is decoupled from appearance, thereby preventing models from reproducing the unnatural look of synthetic videos. Building on this idea, DynaVid adopts a two-stage generation framework: a motion generator first synthesizes motion, and then a motion-guided video generator produces video frames conditioned on that motion. This decoupled formulation enables the model to learn dynamic motion patterns from synthetic data while preserving visual realism from real-world videos. We validate our framework on two challenging scenarios, vigorous human motion generation and extreme camera motion control, where existing datasets are particularly limited. Extensive experiments demonstrate that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.

Problem

• dynamic video generation
• motion controllability
• synthetic motion data
• video diffusion models
• optical flow

Innovation

• synthetic motion data
• optical flow
• video diffusion models
• motion-appearance decoupling
• dynamic video generation