Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music

2026-07-07

AI Summary

Existing music understanding benchmarks suffer from limitations such as static evaluation protocols, high resource demands, absence of authentic perceptual assessment, and lack of cross-modal comparability. This work proposes MusICA-MetaBench, a novel framework that introduces a pedagogy-aligned, on-demand generation paradigm for evaluating multimodal large language models’ musical perception capabilities. Leveraging symbolic inputs like MusicXML provided by users, the framework automatically constructs multimodal multiple-choice questions encompassing audio, sheet music images, and symbolic representations. It integrates structured musical representations, predefined question templates, and statistical reliability analysis to ensure validity. Evaluation on the ChoraleBricks dataset demonstrates the effectiveness of the generated questions and identifies the minimal benchmark size required to support statistically significant model comparisons.

Abstract

Music represents a cornerstone of human culture, existing digitally across diverse modalities, including audio, symbolic encodings (e.g., MIDI, MusicXML), and sheet music. Despite the advancement of Multimodal Large Language Models (MLLMs), current music benchmarks face three major limitations. First, large static benchmarks are resource-intensive to evaluate, and it remains unclear how their results transfer to diverse kinds of music beyond those included in the benchmark. Second, benchmarks claiming to measure "music understanding" often fail to require music perception. Third, they do not support systematic performance comparisons across musical modalities. To overcome these issues, we introduce the Music I Care About Meta-Benchmark (MusICA-MetaBench), a framework that automatically derives on-demand benchmarks directly from user-provided data. By leveraging structured symbolic representations (e.g., MusicXML) and our pre-defined question templates, we build multiple-choice question-answer pairs that probe music perception competencies, aligned with music pedagogy, across audio, music notation images, and symbolic files. We demonstrate our framework with the ChoraleBricks dataset, and experimentally determine benchmark sizes that ensure statistically reliable model comparisons for this setup. By comparing against text-only and white-noise baselines, we show our questions do measure music perception. Ultimately, MusICA-MetaBench represents a significant advancement in the cross-modal assessment of music perception for MLLMs. By proposing a dataset-specific benchmarking paradigm, it enables efficient on-demand evaluation of music perception capabilities.

Problem

• music perception
• multimodal benchmarking
• large language models
• symbolic music representations
• cross-modal evaluation

Innovation

• multimodal benchmarking
• music perception
• on-demand evaluation
• symbolic music representation
• MLLMs