Rubato: Transcribing Piano Music with Timestamps

2026-05-22

AI Summary

This study addresses the challenge of accurately transcribing piano performance recordings by proposing an end-to-end prompt-conditioned encoder-decoder model that directly generates polyphonic music scores with precise timestamps, facilitating rubato visualization, music learning, and analysis. The core innovations include a novel serialized music representation, InterMo, and a unified multi-task prompting framework that overcomes error propagation and representational limitations inherent in traditional cascaded systems. Experimental results demonstrate that the proposed method significantly outperforms state-of-the-art cascaded approaches on timestamped piano transcription—even surpassing systems that use ground-truth MIDI input—and achieves competitive or superior performance compared to specialized models on subtasks such as MIDI note alignment and beat detection.

Abstract

We consider the conversion of musical recordings into human-readable sheet music annotated with timestamps. Such output lets a listener clearly visualize rubato (temporally expressive playing), a learner diagnose ensemble precision and timing choices against the written music, and a musicology scholar compare performance styles across recordings of the same work. We introduce (1) a prompt-conditioned encoder-decoder model, named Rubato, trained to output (2) a new textual representation for polyphonic music, named InterMo, which we designed for compatibility with sequence-to-sequence training. Our experiments demonstrate that Rubato produces timestamped piano sheet music from audio with higher notational accuracy than the best existing approaches, which are based on cascades. We find that even if the cascade is given ground-truth MIDI instead of audio, Rubato performs better, suggesting that the ceiling of existing approaches is primarily representational, not acoustic. Further, because Rubato is trained on several related tasks (with prompts), it competes with or outperforms the best single-task systems on related but simpler tasks like MIDI note grounding and beat/downbeat detection. A demo is available at https://nctamer.github.io/rubato-transcription .

Problem

• music transcription
• timestamped sheet music
• rubato
• piano performance
• polyphonic music

Innovation

• prompt-conditioned encoder-decoder
• InterMo representation
• timestamped music transcription
• polyphonic music
• sequence-to-sequence modeling