Do Foundational Audio Encoders Understand Music Structure?

2025-12-18

AI Summary

Music structure analysis (MSA) lacks systematic empirical evaluation of foundation audio encoders (FAEs), hindering understanding of how pretraining paradigms, training data, and context length affect structural understanding. Method: We conduct a comprehensive benchmark of 11 FAEs on MSA tasks, employing controlled ablation studies, standardized structural annotation benchmarks (e.g., RWC-Pop, SALAMI), feature interpretability analysis, and cross-model attribution. Contribution/Results: We find—firstly—that FAEs pretrained via self-supervised masked language modeling (MLM) significantly outperform supervised baselines in segment-level structural identification; secondly, that pretraining objectives exert greater influence on MSA performance than architectural choices. These results establish FAEs as highly transferable, fine-tuning-free representations for MSA, thereby filling a critical empirical gap in evaluating FAEs’ capacity for music structural understanding.

Technology Category

• Machine Learning: Structured Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In music information retrieval (MIR) research, the use of pretrained foundational audio encoders (FAEs) has recently become a trend. FAEs pretrained on large amounts of music and audio data have been shown to improve performance on MIR tasks such as music tagging and automatic music transcription. However, their use for music structure analysis (MSA) remains underexplored. Although many open-source FAE models are available, only a small subset has been examined for MSA, and the impact of factors such as learning methods, training data, and model context length on MSA performance remains unclear. In this study, we conduct comprehensive experiments on 11 types of FAEs to investigate how these factors affect MSA performance. Our results demonstrate that FAEs using selfsupervised learning with masked language modeling on music data are particularly effective for MSA. These findings pave the way for future research in MSA.

Problem

• Evaluating foundational audio encoders for music structure analysis
• Investigating impact of learning methods and training data on performance
• Identifying effective self-supervised learning approaches for MSA

Innovation

• Self-supervised learning with masked language modeling on music data
• Comprehensive experiments on 11 types of foundational audio encoders
• Investigating impact of learning methods, training data, and context length