Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder

2025-07-02

AI Summary

This work addresses the challenge of jointly modeling musical structure and expressive performance in classical piano music generation. To this end, we propose a unified composition-and-performance generation framework. Methodologically, we introduce Expressive Compound Word (ECW), a novel musical representation that disentangles structural elements (e.g., pitch, duration) from expressive dimensions (e.g., velocity, tempo). We further design XMVAE, a dual-branch variational autoencoder: one branch employs VQ-VAE to model structural tokens, while the other adopts VAE for expressive features; both are enhanced with multi-scale encoders, an orthogonal Transformer decoder, and a compound-token decoding mechanism to enable fine-grained, synchronized generation. Experiments demonstrate that our approach significantly outperforms state-of-the-art models in both objective metrics and subjective evaluations, yielding performances with superior expressivity, coherence, and musicality—further improved via pretraining.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The creativity of classical music arises not only from composers who craft the musical sheets but also from performers who interpret the static notations with expressive nuances. This paper addresses the challenge of generating classical piano performances from scratch, aiming to emulate the dual roles of composer and pianist in the creative process. We introduce the Expressive Compound Word (ECP) representation, which effectively captures both the metrical structure and expressive nuances of classical performances. Building on this, we propose the Expressive Music Variational AutoEncoder (XMVAE), a model featuring two branches: a Vector Quantized Variational AutoEncoder (VQ-VAE) branch that generates score-related content, representing the Composer, and a vanilla VAE branch that produces expressive details, fulfilling the role of Pianist. These branches are jointly trained with similar Seq2Seq architectures, leveraging a multiscale encoder to capture beat-level contextual information and an orthogonal Transformer decoder for efficient compound tokens decoding. Both objective and subjective evaluations demonstrate that XMVAE generates classical performances with superior musical quality compared to state-of-the-art models. Furthermore, pretraining the Composer branch on extra musical score datasets contribute to a significant performance gain.

Problem

• Generate classical piano performances from scratch
• Capture metrical structure and expressive nuances
• Emulate composer and pianist roles in music creation

Innovation

• Expressive Compound Word representation captures performance nuances
• XMVAE model combines VQ-VAE and VAE branches
• Multiscale encoder and Transformer decoder enhance generation