Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models

2026-06-30

AI Summary

While existing music audio–language models achieve strong performance on instrument-related question answering tasks, their genuine audio grounding capabilities remain questionable. This work introduces the first multi-axis diagnostic benchmark built upon OpenMIC, systematically evaluating models’ fine-grained grounding through challenging scenarios encompassing genre debiasing, confusion-prone instrument discrimination, long-context reasoning, and temporal localization. Experimental results reveal that high binary QA accuracy often masks underlying issues such as positional bias, misclassification of acoustically similar instruments, and temporal response distortions. These findings demonstrate that conventional binary QA metrics are insufficient to capture true model competence, advocating for a paradigm shift toward fine-grained, diagnostic evaluation frameworks in audio–language grounding research.

Abstract

Recent music audio-language models achieve high accuracy on instrument question-answering benchmarks, but it remains unclear whether this reflects robust audio grounding or benchmark-specific shortcuts. In this paper, we introduce an OpenMIC-derived diagnostic benchmark sequence for instrument grounding in music audio-language models, extending binary instrument-presence QA to genre-prior-reduced examples, confusable instrument discrimination, longer audio context, and temporal localization. Across these settings, high binary QA accuracy often fails to predict model behavior: models can exhibit option-position bias, confusable-instrument errors, and temporal response bias. These results suggest that instrument grounding should be evaluated with multi-axis diagnostic benchmarks rather than a single aggregate accuracy.

Problem

• instrument grounding
• music audio-language models
• diagnostic benchmark
• binary QA
• audio grounding

Innovation

• instrument grounding
• audio-language models
• diagnostic benchmark
• confusable instrument discrimination
• temporal localization