Unifying Score and Performance for Fine-Grained Music Understanding in Audio-Language Models

2026-09-09

AI Summary

研究通过引入MuNo-SP表示法和自动训练数据生成管道，解决了现有音频-语言模型在细粒度音乐理解上的不足，提高了模型对音乐表现细节的学习能力。

Abstract

Large audio language models (LALMs) have shown promising progress in broad music-understanding tasks such as tagging, retrieval, and captioning. Music understanding that requires finer hearing over both the content and how it is realized within a performance through dynamics, phrasing, articulation, time, and other performance techniques, however, remains at an earlier stage. Existing audio-language model (ALM) training pipelines typically rely on coarse, weakly grounded captions and therefore provide little support for learning these subtle nuances in music, limiting their ability to serve real-world applications in education or artistic practice. We therefore introduce MuNo-SP (Music Notation unifying Score and Performance), a text-based representation that jointly encodes score content and performance information. Building on MuNo-SP, we develop an automatic training-data generation pipeline that uses aligned scores and performances to produce long-form auditory analyses and musically informed question-answer pairs. We use this pipeline to construct MAESTROCaps, a classical piano dataset comprising 148 long-form performance analyses and 31,080 question-answer pairs derived from 148 aligned score-performance pairs. In a human evaluation, MuNo-SP analyses were preferred by majority vote over MIDI-only analyses for eight of nine excerpts. MuNo-SP also performed strongly on a benchmark of score-performance understanding, suggesting that integrating score and performance information enables more reliable and musically informative LALM supervision than a MIDI-only baseline.

Problem

• audio-language model
• music understanding
• performance nuances
• education and artistic practice
• score and performance

Innovation

• MuNo-SP
• Score and Performance Unification
• Automatic Training Data Generation
• MAESTROCaps