MUSIC: Learning Muscle-Driven Dexterous Hand Control

2026-04-26

AI Summary

This work proposes a physics-based, muscle-driven hierarchical control framework that achieves, for the first time, physiologically plausible piano performance synthesis across diverse musical pieces. Addressing the challenge of dexterous hands accurately playing unseen scores, the method integrates high-frequency muscle-level actuation with low-frequency coordination in a structured latent space. It first trains a single-hand policy via reinforcement learning to track reference motions, then distills this policy into a variational autoencoder to construct a structured latent representation. Finally, piece-specific policies are trained within this latent space to coordinate bimanual movements according to musical note events from the score. Experiments demonstrate that the system achieves high-fidelity keypress accuracy and inter-hand coordination across multiple musical styles, establishing a new state of the art in physics-based dexterous control, with synthesized muscle activation patterns closely matching human electromyography recordings.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Intelligent Robots: Behavior Learning & Control

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present a data-driven approach for physics-based, muscle-driven dexterous control that enables musculoskeletal hands to perform precise piano playing for novel pieces of music outside the reference dataset. Our approach combines high-frequency muscle-level control with low-frequency latent-space coordination in a hierarchical architecture. At the low level, general single-hand policies are trained via reinforcement learning to generate dynamic muscle-tendon activations while tracking trajectories from a large reference motion dataset. The resulting tracking policies are then distilled into variational autoencoder (VAE) models, yielding smooth and structured latent spaces that abstract away low-level muscle dynamics. For the high level, we train piece-specific policies to operate in this latent space, coordinating bimanual motions based on specific goals, denoted by note events extracted from given musical scores, to synthesize performances beyond the reference data. In addition, we present an enhanced musculoskeletal hand model that supports fine control of fingers for accurate low-level motion tracking and diverse high-level motion synthesis. We evaluate the control pipeline of our approach on a diverse piano repertoire spanning multiple musical styles and technical demands. Results demonstrate that our approach can synthesize coordinated bimanual motions with accurate key presses, and achieve the state-of-the-art performance of piano playing in physics-based dexterous control. We also show that our musculoskeletal hand model demonstrates superior biomechanical stability and tracking precision compared to the existing model, and validate that our musculoskeletal hand model and muscle-driven controller can generate physiologically plausible activation patterns that align with human electromyography (EMG) recordings.

Problem

• dexterous hand control
• muscle-driven
• physics-based simulation
• bimanual coordination
• piano playing

Innovation

• muscle-driven control
• hierarchical reinforcement learning
• variational autoencoder
• musculoskeletal hand model
• dexterous manipulation