Enforcing Human-like Kinematics in Dexterous Piano Playing via Adversarial Posture Regularization

2026-06-22

AI Summary

This work addresses the tendency of high-degree-of-freedom dexterous hands in reinforcement learning to produce unnatural postures and joint overextension, lacking human-like kinematic characteristics. To enhance the naturalness of hand motions without requiring costly aligned demonstration data, the authors propose Adversarial Posture Regularization (APR), which leverages only a small set of unaligned human piano-playing videos. Through adversarial learning, APR aligns the policy’s output posture distribution with human motion priors. Integrated with reinforcement learning, motion retargeting, and consumer-grade VR hand tracking, the method significantly outperforms existing approaches on three human-likeness metrics—cPSI, BSE, and FAC—as well as visual quality. The study also releases a hand motion dataset collected using the Meta Quest 3.

Abstract

Reinforcement learning can train bimanual dexterous hands to play piano in physics simulation with high note accuracy, but for high-DoF dexterous hands, relying solely on task rewards or IK inversion often leads to unnatural postures and joint overextension. We propose \textit{Adversarial Posture Regularization (APR)}. It avoids expensive, song-aligned expert demonstration data and instead uses a small amount of casual human playing data. By matching the distribution of the posture of the policy with the human prior through an adversarial objective, APR encourages more human-like hand shapes. Meanwhile, we collect and release unstructured hand motion data of piano playing using a consumer-grade Meta Quest 3, and retarget the key motion information to the Shadow Hand. Finally, we achieve significantly better performance than prior methods on all three human-likeness metrics (cPSI, BSE, and FAC) as well as in visual quality. Project repository: https://github.com/APRProject/APRPianist.

Problem

• dexterous manipulation
• human-like kinematics
• piano playing
• unnatural postures
• joint overextension

Innovation

• Adversarial Posture Regularization
• dexterous manipulation
• human-like kinematics
• reinforcement learning
• motion retargeting