LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

2026-07-06

AI Summary

This work addresses the challenge of jointly extracting symbolic notation and semantic knowledge from complex, long-form musical scores containing embedded textual annotations. We propose the first neural optical music recognition (OMR) pipeline capable of system-level sequential processing, integrating system-level score segmentation with an autoregressive vision-language model to process staves in reading order. This approach enables unified modeling of local note-level details and global structural context, producing complete symbolic transcriptions that preserve embedded text such as titles and annotations. Experimental results demonstrate that our method achieves new state-of-the-art performance across multiple benchmarks, significantly outperforming existing approaches. Furthermore, we show that high-quality symbolic transcriptions generated by our pipeline substantially enhance large language models’ ability to semantically interpret music documents.

Abstract

We propose a novel pipeline, Legato 2, for extracting symbolic notation and semantic knowledge from images of sheet music. Legato 2 features the first large-scale neural model for optical music recognition (OMR) to operate sequentially on a system-by-system basis, following the horizontal lines of notation as they are read on the page, rather than treating the page as an undifferentiated image, enabling better scaling to arbitrarily long inputs. It is also the first OMR model capable of generating symbolic transcriptions that include embedded textual content, such as titles and annotations. The pipeline combines system-level segmentation with an autoregressive vision-LM to capture both local notation details and score structure. Across multiple datasets, Legato 2 consistently outperforms prior state of the art. We also show that symbolic transcriptions complement visual inputs for frontier language models, improving their interpretation of dense musical documents. Legato 2 establishes new state-of-the-art performance in both OMR and downstream sheet music understanding.

Problem

• optical music recognition
• sheet music understanding
• symbolic transcription
• multimodal recognition
• music notation

Innovation

• optical music recognition
• multimodal understanding
• vision-language model
• symbolic transcription
• system-level segmentation