Toward Rich Video Human-Motion2D Generation

2025-06-17

AI Summary

To address data scarcity and the challenge of modeling inter-personal dynamics in multi-role interactive 2D human motion video generation, this work introduces Motion2D-Video-150K—the first large-scale, fine-grained text-annotated dataset comprising 150K motion sequences. We propose RVHM2D, a novel framework establishing the first dual-character interactive motion generation paradigm. It features a dual-path text encoder integrating CLIP-L/B and T5-XXL to jointly capture global and local semantics, and adopts a FID-driven two-stage training strategy: diffusion-based pretraining followed by reinforcement-based fine-tuning. Evaluated on the Motion2D-Video-150K benchmark, RVHM2D achieves state-of-the-art performance on both single- and two-person motion generation tasks, significantly improving text-motion alignment and motion realism.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Generating realistic and controllable human motions, particularly those involving rich multi-character interactions, remains a significant challenge due to data scarcity and the complexities of modeling inter-personal dynamics. To address these limitations, we first introduce a new large-scale rich video human motion 2D dataset (Motion2D-Video-150K) comprising 150,000 video sequences. Motion2D-Video-150K features a balanced distribution of diverse single-character and, crucially, double-character interactive actions, each paired with detailed textual descriptions. Building upon this dataset, we propose a novel diffusion-based rich video human motion2D generation (RVHM2D) model. RVHM2D incorporates an enhanced textual conditioning mechanism utilizing either dual text encoders (CLIP-L/B) or T5-XXL with both global and local features. We devise a two-stage training strategy: the model is first trained with a standard diffusion objective, and then fine-tuned using reinforcement learning with an FID-based reward to further enhance motion realism and text alignment. Extensive experiments demonstrate that RVHM2D achieves leading performance on the Motion2D-Video-150K benchmark in generating both single and interactive double-character scenarios.

Problem

• Generating realistic multi-character interactive human motions
• Addressing data scarcity in human motion modeling
• Enhancing motion realism and text alignment

Innovation

• Large-scale Motion2D-Video-150K dataset creation
• Diffusion-based RVHM2D model with enhanced text conditioning
• Two-stage training with diffusion and RL fine-tuning