MuSViT: A Foundation Vision Model for Sheet Music Representation

2026-06-30

AI Summary

This work addresses the lack of dedicated foundational vision models for music score understanding, which has limited performance across diverse downstream tasks. We propose MuSViT—the first foundation vision model specifically designed for music score representation—built upon the Vision Transformer architecture and pretrained via masked autoencoding on 9.7 million pages of IMSLP scores. Our approach employs a two-stage curriculum strategy, beginning with synthetic scores for pre-warming followed by training on real scores. MuSViT’s learned representation space directly encodes symbolic musical structures, enabling it to significantly outperform general-purpose vision models even with a frozen encoder. When fine-tuned, MuSViT achieves state-of-the-art results across four tasks: full-page and staff-level recognition, music symbol detection, and difficulty classification, demonstrating its effectiveness as a universal backbone for music score understanding.

Abstract

Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.

Problem

• sheet music
• foundation model
• visual representation
• music notation
• symbolic structure

Innovation

• foundation model
• sheet music understanding
• Vision Transformer
• Masked Autoencoder
• curriculum pre-training