BASS: Benchmarking Audio LMs for Musical Structure and Semantic Reasoning

2026-02-03

AI Summary

This work addresses the limited capability of existing audio language models in high-level music understanding tasks—such as structural segmentation, musicological analysis, and artist collaboration—due to their insufficient capacity for deep reasoning about musical structure and semantics. To bridge this gap, the authors introduce BASS, a comprehensive benchmark comprising 12 tasks and 2,658 questions spanning 1,993 songs (138 hours of diverse music genres), which uniquely integrates musical structure and semantic reasoning into a unified evaluation framework. Leveraging a human-curated, high-quality multitask question-answering dataset, the study conducts a systematic evaluation of 14 state-of-the-art multimodal language models. Results reveal that while models perform reasonably well on lyric transcription, they exhibit significant limitations in higher-order tasks, largely due to overreliance on linguistic priors at the expense of intrinsic musical features. This work establishes the first dedicated benchmark for assessing music cognition capabilities in audio language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Music understanding is a complex task that often requires reasoning over both structural and semantic elements of audio. We introduce BASS, designed to evaluate music understanding and reasoning in audio language models across four broad categories: structural segmentation, lyric transcription, musicological analysis, and artist collaboration. BASS comprises 2658 questions spanning 12 tasks, 1993 unique songs and covering over 138 hours of music from a wide range of genres and tracks, crafted to assess musicological knowledge and reasoning in real-world scenarios. We evaluate 14 open-source and frontier multimodal LMs, finding that even state-of-the-art models struggle on higher-level reasoning tasks such as structural segmentation and artist collaboration, while performing best on lyric transcription. Our analysis reveals that current models leverage linguistic priors effectively but remain limited in reasoning over musical structure, vocal, and musicological attributes. BASS provides an evaluation framework with widespread applications in music recommendation and search and has the potential to guide the development of audio LMs.

Problem

• music understanding
• audio language models
• structural reasoning
• semantic reasoning
• musicological analysis

Innovation

• audio language models
• music understanding
• structural segmentation
• musicological reasoning
• multimodal evaluation benchmark