End-to-End Full-Page Optical Music Recognition for Pianoform Sheet Music

2024-05-20

AI Summary

Existing end-to-end, full-page optical music recognition (OMR) methods rely on multi-stage pipelines and dedicated layout analysis, limiting generalization. This work introduces the first truly end-to-end full-page piano score OMR system, directly mapping raw page images to structured MusicXML representations. Our approach features: (1) the first full-page, end-to-end OMR architecture integrating convolutional feature extraction with an autoregressive Transformer decoder; (2) a curriculum learning–driven progressive synthetic data generation and training paradigm; and (3) zero-shot performance on real piano scores surpassing leading commercial OMR software. Experiments demonstrate state-of-the-art (SOTA) accuracy on both synthetic data and two real-world benchmarks—achieving significant improvements over existing tools under both zero-shot and fine-tuned settings. The proposed method eliminates hand-crafted heuristics and stage-wise dependencies, enabling robust, unified transcription of entire musical pages without intermediate structural assumptions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Optical Music Recognition (OMR) has made significant progress since its inception, with various approaches now capable of accurately transcribing music scores into digital formats. Despite these advancements, most so-called emph{end-to-end} OMR approaches still rely on multi-stage processing pipelines for transcribing full-page score images, which introduces several limitations that hinder the full potential of the field. In this paper, we present the first truly end-to-end approach for page-level OMR. Our system, which combines convolutional layers with autoregressive Transformers, processes an entire music score page and outputs a complete transcription in a music encoding format. This is made possible by both the architecture and the training procedure, which utilizes curriculum learning through incremental synthetic data generation. We evaluate the proposed system using pianoform corpora. This evaluation is conducted first in a controlled scenario with synthetic data, and subsequently against two real-world corpora of varying conditions. Our approach is compared with leading commercial OMR software. The results demonstrate that our system not only successfully transcribes full-page music scores but also outperforms the commercial tool in both zero-shot settings and after fine-tuning with the target domain, representing a significant contribution to the field of OMR.

Problem

• Transcribing full-page pianoform sheet music end-to-end
• Eliminating multi-stage pipelines for music score recognition
• Handling complex layouts without dedicated layout analysis

Innovation

• End-to-end full-page OMR with Transformers
• Curriculum learning via synthetic data
• Outperforms commercial OMR software