Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesis

2025-04-14

AI Summary

To address the challenge of modeling piano-playing hand motions—requiring both individual hand characteristics and precise inter-hand coordination—this paper proposes the first audio-driven, high-fidelity 3D hand motion synthesis method for duet piano performance. Methodologically, we introduce a dual-stream diffusion architecture augmented with a Hand Coordination Asymmetric Attention (HCAA) mechanism: dual noise initialization enables disentangled hand motion modeling while suppressing common-mode noise; an audio-feature-driven hierarchical generation framework integrates a position prediction network with a position-aware dual-stream diffusion model, incorporating shared positional conditioning and cross-stream HCAA interaction. Experiments demonstrate that our approach consistently outperforms existing state-of-the-art methods across key metrics—including motion naturalness, inter-hand temporal alignment accuracy, and coordination fidelity—yielding significant improvements in both generative quality and physical plausibility.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Automating the synthesis of coordinated bimanual piano performances poses significant challenges, particularly in capturing the intricate choreography between the hands while preserving their distinct kinematic signatures. In this paper, we propose a dual-stream neural framework designed to generate synchronized hand gestures for piano playing from audio input, addressing the critical challenge of modeling both hand independence and coordination. Our framework introduces two key innovations: (i) a decoupled diffusion-based generation framework that independently models each hand's motion via dual-noise initialization, sampling distinct latent noise for each while leveraging a shared positional condition, and (ii) a Hand-Coordinated Asymmetric Attention (HCAA) mechanism suppresses symmetric (common-mode) noise to highlight asymmetric hand-specific features, while adaptively enhancing inter-hand coordination during denoising. The system operates hierarchically: it first predicts 3D hand positions from audio features and then generates joint angles through position-aware diffusion models, where parallel denoising streams interact via HCAA. Comprehensive evaluations demonstrate that our framework outperforms existing state-of-the-art methods across multiple metrics.

Problem

• Synthesizing coordinated bimanual piano hand motions from audio
• Modeling hand independence and coordination in piano performances
• Generating synchronized 3D hand positions and joint angles

Innovation

• Dual-stream diffusion model for hand motion
• Hand-Coordinated Asymmetric Attention mechanism
• Hierarchical audio-to-position-to-angle synthesis