Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades

2026-03-09

AI Summary

Existing video generation models struggle to precisely control the temporal details of complex human motions through text alone, while explicit skeleton-based control requires users to provide lengthy pose sequences, which is labor-intensive and costly. To address this, this work proposes a two-stage cascaded framework: first, an autoregressive text-to-2D-pose model generates motion sequences from textual descriptions; then, a pose-conditioned video diffusion model synthesizes high-quality videos by combining these pose sequences with a reference image. The approach introduces a novel DINO-ALF multi-level reference encoding mechanism to maintain appearance consistency under large pose variations and constructs the first synthetic dataset comprising 2,000 highly controllable acrobatic motion clips. Experiments demonstrate that the proposed method significantly outperforms existing approaches in both pose generation and video synthesis on the newly curated dataset and the Motion-X Fitness benchmark.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Generating videos of complex human motions such as flips, cartwheels, and martial arts remains challenging for current video diffusion models. Text-only conditioning is temporally ambiguous for fine-grained motion control, while explicit pose-based controls, though effective, require users to provide complete skeleton sequences that are costly to produce for long and dynamic actions. We propose a two-stage cascaded framework that addresses both limitations. First, an autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions by predicting each joint conditioned on previously generated poses. This design captures long-range temporal dependencies and inter-joint coordination required for complex motions. Second, a pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence. It employs DINO-ALF (Adaptive Layer Fusion), a multi-level reference encoder that preserves appearance and clothing details under large pose changes and self-occlusions. To address the lack of publicly available datasets for complex human motion video generation, we introduce a Blender-based synthetic dataset containing 2,000 videos with diverse characters performing acrobatic and stunt-like motions. The dataset provides full control over appearance, motion, and environment. It fills an important gap because existing benchmarks significantly under-represent acrobatic motions while web-collected datasets raise copyright and privacy concerns. Experiments on our synthetic dataset and the Motion-X Fitness benchmark show that our text-to-skeleton model outperforms prior methods on FID, R-precision, and motion diversity. Our pose-to-video model also achieves the best results among all compared methods on VBench metrics for temporal consistency, motion smoothness, and subject preservation.

Problem

• complex human motion
• video generation
• text-to-skeleton
• motion control
• pose conditioning

Innovation

• text-to-skeleton
• cascaded generation
• DINO-ALF
• synthetic motion dataset
• complex human motion