XmoPipe: A Pipeline for Large-Scale In-the-Wild Human Motion Dataset Construction

2026-06-17

AI Summary

Traditional marker-based motion capture is costly and yields limited-scale, low-diversity datasets, hindering large-scale human motion modeling. This work proposes the first end-to-end scalable pipeline that automatically extracts 3D body and facial motions from in-the-wild internet videos and generates corresponding semantic textual descriptions, enabling the construction of high-quality, annotated motion datasets without controlled environments. By integrating monocular motion capture, video–language understanding, 3D pose and facial action estimation, and text generation, the method substantially enhances the flexibility and scalability of motion data acquisition. Motion reconstruction and generation models trained on this dataset achieve performance comparable to those trained on conventional motion capture data and demonstrate strong cross-dataset generalization capabilities.

Abstract

Large-scale human motion datasets are essential for training robust motion models for analysis, synthesis, and understanding. While marker-based motion capture provides precise data, it is costly and limited in scale and diversity. Recent advances in monocular motion capture and video-language understanding open the way to extract plausible motion from unconstrained online videos. We present a scalable pipeline for constructing in-the-wild human motion datasets. From a few keywords, the system retrieves videos, extracts 3D body and facial motion, and generates high-level textual descriptions. The pipeline is flexible, enabling targeted collection of various motions, multi-person interactions, or expressive behaviors. We demonstrate its quality by training motion reconstruction and motion generation models, showing performance comparable to models trained on traditional motion capture datasets and strong cross-dataset generalization.

Problem

• human motion dataset
• in-the-wild
• large-scale
• motion capture
• 3D motion

Innovation

• monocular motion capture
• in-the-wild dataset
• video-language understanding
• scalable pipeline
• 3D human motion