Problem
Research questions and friction points this paper is trying to address.
wearable motion capture
arbitrary sensor configurations
full-body motion reconstruction
consumer-grade sensors
multi-modal sensing
Towards Real-World Wearable Motion Reconstruction
🤖 AI Summary
This work addresses the challenge of achieving generalizable and robust full-body motion reconstruction using arbitrary combinations of lightweight wearable sensors—such as smartphones, smartwatches, eyewear, and insoles—by introducing a large-scale, multimodal, synchronized dataset encompassing 50 daily activities. The authors propose WHIP, a generative model capable of reconstructing physically plausible full-body motions from any subset of available sensor inputs, effectively handling missing modalities. For the first time, the study systematically quantifies the complementary nature of different wearable sensors and demonstrates the model’s high fidelity and strong generalization across diverse real-world scenarios.
📝 Abstract
The modern-day surge in popularity of wearable devices poses a fundamentally unique motion capture problem: reconstructing full-body movement from any set of sensing hardware worn at a given moment. Yet, most research efforts assume fixed sensor configurations (e.g. IMU suits or HMD-centric rigs) and cannot generalize across them. In contrast, we argue that motion capture should prioritize unobtrusive and lightweight devices such as smartphones, smartwatches, smart glasses, and smart insoles, and study the interplay between them. To this end, we make three contributions. First, we present a large-scale multi-modal dataset synchronizing these consumer-grade sensors with ground-truth 3D motion, spanning 50 diverse activities including everyday tasks, sports, and social interactions. Second, we propose WHIP, a baseline generative model that reconstructs motion from arbitrary subsets of available sensors, robustly handling missing modalities and producing physically plausible motions. Third, we conduct a systematic study of sensor complementarity, quantifying how different modalities complement one another. Code and dataset are available at https://vcai.mpi-inf.mpg.de/projects/WHIP/
Innovation
Methods, ideas, or system contributions that make the work stand out.
wearable motion capture
multi-modal sensing
generative motion reconstruction
sensor complementarity
consumer-grade IMUs
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
