Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets

2025-09-08

AI Summary

Current automatic music annotation lacks standardized evaluation benchmarks grounded in expert musicological annotation (e.g., MGPHot), and general-purpose tag datasets are inadequate for deep semantic assessment. Method: We introduce the first benchmark for automatic music annotation based on expert musicological labeling: it integrates MGPHot expert annotations with reproducible YouTube audio sources, provides standardized train/validation/test splits, and publicly releases precomputed features from seven state-of-the-art audio representation models (e.g., CLAP, Jukebox). Contribution/Results: This benchmark pioneers the integration of domain-expert musicological annotation into automatic annotation evaluation. Experiments demonstrate that expert labels significantly outperform generic tags in semantic granularity, label consistency, and model evaluation robustness. They critically enhance annotation quality and assessment reliability, establishing a finer-grained, more comparable, and fully reproducible evaluation paradigm for music understanding models.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Music autotagging aims to automatically assign descriptive tags, such as genre, mood, or instrumentation, to audio recordings. Due to its challenges, diversity of semantic descriptions, and practical value in various applications, it has become a common downstream task for evaluating the performance of general-purpose music representations learned from audio data. We introduce a new benchmarking dataset based on the recently published MGPHot dataset, which includes expert musicological annotations, allowing for additional insights and comparisons with results obtained on common generic tag datasets. While MGPHot annotations have been shown to be useful for computational musicology, the original dataset neither includes audio nor provides evaluation setups for its use as a standardized autotagging benchmark. To address this, we provide a curated set of YouTube URLs with retrievable audio, and propose a train/val/test split for standardized evaluation, and precomputed representations for seven state-of-the-art models. Using these resources, we evaluated these models in MGPHot and standard reference tag datasets, highlighting key differences between expert and generic tag annotations. Altogether, our contributions provide a more advanced benchmarking framework for future research in music understanding.

Problem

• Benchmarking music autotagging with expert vs generic annotations
• Providing standardized evaluation framework with audio data
• Comparing model performance across different annotation types

Innovation

• Expert annotations from MGPHot dataset
• Curated YouTube URLs with retrievable audio
• Precomputed representations for seven models