MoCo: Motion-Consistent Human Video Generation via Structure-Appearance Decoupling

2025-08-24

AI Summary

Text-to-full-body human video generation suffers from motion discontinuity, physically implausible dynamics, and structural distortions—primarily due to existing models’ overemphasis on appearance fidelity at the expense of explicit motion modeling and scarcity of high-quality, full-body motion data. This paper proposes a structure-appearance disentanglement framework: a 3D structure generator first produces physically plausible full-body motion sequences, which are then refined by a diffusion-based appearance synthesizer. To enable precise temporal control from sparse keypoints, we introduce a human-perceptual dynamic control module and impose dense pose tracking constraints. Furthermore, we construct the first large-scale, multi-view, long-duration full-body motion dataset. Experiments demonstrate that our method significantly outperforms prior approaches in motion coherence, physical plausibility, and visual realism, enabling robust generation of complex, diverse, and long-horizon full-body motions.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Generating human videos with consistent motion from text prompts remains a significant challenge, particularly for whole-body or long-range motion. Existing video generation models prioritize appearance fidelity, resulting in unrealistic or physically implausible human movements with poor structural coherence. Additionally, most existing human video datasets primarily focus on facial or upper-body motions, or consist of vertically oriented dance videos, limiting the scope of corresponding generation methods to simple movements. To overcome these challenges, we propose MoCo, which decouples the process of human video generation into two components: structure generation and appearance generation. Specifically, our method first employs an efficient 3D structure generator to produce a human motion sequence from a text prompt. The remaining video appearance is then synthesized under the guidance of the generated structural sequence. To improve fine-grained control over sparse human structures, we introduce Human-Aware Dynamic Control modules and integrate dense tracking constraints during training. Furthermore, recognizing the limitations of existing datasets, we construct a large-scale whole-body human video dataset featuring complex and diverse motions. Extensive experiments demonstrate that MoCo outperforms existing approaches in generating realistic and structurally coherent human videos.

Problem

• Generating consistent human motion from text prompts
• Overcoming unrealistic movements in video generation
• Addressing limited whole-body motion datasets

Innovation

• Structure-appearance decoupling for human video generation
• 3D structure generator for motion sequences
• Human-aware dynamic control with tracking constraints