Switch-JustDance: Benchmarking Whole Body Motion Tracking Policies Using a Commercial Console Game

2025-11-22

AI Summary

Existing whole-body motion tracking benchmarks for humanoid/legged robots lack standardized, real-world evaluation protocols—particularly for direct, reproducible comparison against human performance. Method: We propose the first low-cost, cross-platform benchmarking framework for whole-body control, leveraging the commercial motion-sensing game *Just Dance*. It employs real-time motion streaming, pose reconstruction, and kinematic retargeting to map human motion capture data onto robot kinematic models and execute dance sequences on physical robots. Control performance is quantified using the game’s built-in scoring system, enabling fair, task-, environment-, and metric-aligned human-robot comparison. Contribution/Results: This work pioneers the use of consumer-grade motion games in robotics benchmarking, empirically validating the framework’s reliability and sensitivity. We conduct a systematic benchmark of three state-of-the-art controllers on a real humanoid robot, revealing critical differences in dynamic coordination and execution accuracy.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Recent advances in whole-body robot control have enabled humanoid and legged robots to perform increasingly agile and coordinated motions. However, standardized benchmarks for evaluating these capabilities in real-world settings, and in direct comparison to humans, remain scarce. Existing evaluations often rely on pre-collected human motion datasets or simulation-based experiments, which limit reproducibility, overlook hardware factors, and hinder fair human-robot comparisons. We present Switch-JustDance, a low-cost and reproducible benchmarking pipeline that leverages motion-sensing console games, Just Dance on the Nintendo Switch, to evaluate robot whole-body control. Using Just Dance on the Nintendo Switch as a representative platform, Switch-JustDance converts in-game choreography into robot-executable motions through streaming, motion reconstruction, and motion retargeting modules and enables users to evaluate controller performance through the game's built-in scoring system. We first validate the evaluation properties of Just Dance, analyzing its reliability, validity, sensitivity, and potential sources of bias. Our results show that the platform provides consistent and interpretable performance measures, making it a suitable tool for benchmarking embodied AI. Building on this foundation, we benchmark three state-of-the-art humanoid whole-body controllers on hardware and provide insights into their relative strengths and limitations.

Problem

• Benchmarking whole-body robot motion tracking policies
• Evaluating robot control using commercial console games
• Comparing humanoid robot performance with human dance

Innovation

• Leveraging commercial console games for robot benchmarking
• Converting game choreography into robot-executable motions
• Using built-in scoring system for controller evaluation