Local Multimodal Music Alignment from Global Supervision

2026-07-10

AI Summary

This work proposes FuSiLi, a novel method that achieves high-quality fine-grained cross-modal alignment between audio and sheet music images without requiring local annotations, operating solely under coarse-grained global supervision such as paired audio–sheet music segments. Leveraging pretrained CLIP and CLAP encoders, FuSiLi establishes local correspondences between audio frames and sheet music image patches through a Sinkhorn-based soft alignment mechanism. It further introduces a hybrid contrastive learning objective that jointly optimizes local alignment and global cross-modal retrieval. Experimental results demonstrate that FuSiLi significantly outperforms existing baselines on frame-level local alignment tasks while maintaining competitive performance in cross-modal retrieval.

Abstract

Understanding music requires understanding localized relationships across data modalities, e.g., how time in performance audio maps onto position in a score image. Yet supervision for such local correspondences is difficult to obtain-in practice, we often only have access to coarser global supervision like paired segments of audio and images. To address this gap, we propose FuSiLi (Fused Sinkhorn-Localized Similarity), a similarity score for multimodal contrastive learning operating directly on local image patch and audio frame features via Sinkhorn-based soft alignment. We show that FuSiLi (i) effectively learns local relationships, (ii) requires only global supervision, and (iii) retains the global alignment capabilities of conventional contrastive approaches. We fine-tune pretrained CLIP and CLAP encoders on pairs of raw sheet music images and audio using a hybrid contrastive objective combining FuSiLi with conventional global similarity. We evaluate on cross-modal retrieval and frame-level alignment tasks against a range of global and local baselines, showing that our approach outperforms them on local alignment while remaining competitive on retrieval.

Problem

• local alignment
• multimodal music
• global supervision
• cross-modal correspondence
• fine-grained alignment

Innovation

• FuSiLi
• Sinkhorn alignment
• multimodal contrastive learning
• local alignment
• global supervision