Exploring the Role of Synthetic Data Augmentation in Controllable Human-Centric Video Generation

2026-04-23

AI Summary

Controllable human video generation is hindered by the scarcity of real-world data, particularly for rare identities and complex motion scenarios. This work proposes a unified diffusion-based framework that systematically investigates, for the first time, the synergistic mechanisms between synthetic and real data in human-centric video generation. It reveals their complementary roles and introduces an efficient synthetic sample selection strategy to enhance training. The proposed approach significantly improves motion realism, temporal coherence, and identity fidelity in generated videos, establishing a new paradigm for building data-efficient and generalizable controllable video generation models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Controllable human video generation aims to produce realistic videos of humans with explicitly guided motions and appearances,serving as a foundation for digital humans, animation, and embodied AI.However, the scarcity of largescale, diverse, and privacy safe human video datasets poses a major bottleneck, especially for rare identities and complex actions.Synthetic data provides a scalable and controllable alternative,yet its actual contribution to generative modeling remains underexplored due to the persistent Sim2Real gap.In this work,we systematically investigate the impact of synthetic data on controllable human video generation. We propose a diffusion-based framework that enables fine-grained control over appearance and motion while providing a unfied testbed to analyze how synthetic data interacts with real world data during training. Through extensive experiments, we reveal the complementary roles of synthetic and real data and demonstrate possible methods for efficiently selecting synthetic samples to enhance motion realism,temporal consistency,and identity preservation.Our study offers the first comprehensive exploration of synthetic data's role in human-centric video synthesis and provides practical insights for building data-efficient and generalizable generative models.

Problem

• controllable human video generation
• synthetic data augmentation
• data scarcity
• Sim2Real gap
• human-centric video synthesis

Innovation

• synthetic data augmentation
• controllable video generation
• diffusion models
• human-centric synthesis
• Sim2Real gap