Lead Instrument Detection from Multitrack Music

2025-03-05

AI Summary

Fine-grained detection and generalization of dominant instruments in multi-track music remain challenging due to label scarcity and acoustic variability across instruments and domains. Method: We introduce the first expert-annotated multi-track dataset and propose a novel framework integrating self-supervised pretraining with track-level, frame-wise attention. A dynamic track attention module selectively aggregates features based on auditory salience; auxiliary track classification and permutation-based data augmentation further enhance cross-instrument and cross-domain robustness. Contribution/Results: Experiments demonstrate substantial improvements over SVM and CRNN baselines on unseen instruments and out-of-distribution test sets. Our approach achieves high accuracy, strong generalization, and interpretable predictions—enabling transparent identification of lead instruments per track. This work establishes a new paradigm for multi-track music analysis, advancing both methodological rigor and practical applicability in instrument-aware music understanding.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Tracking, profiling, and countermeasures against them
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Prior approaches to lead instrument detection primarily analyze mixture audio, limited to coarse classifications and lacking generalization ability. This paper presents a novel approach to lead instrument detection in multitrack music audio by crafting expertly annotated datasets and designing a novel framework that integrates a self-supervised learning model with a track-wise, frame-level attention-based classifier. This attention mechanism dynamically extracts and aggregates track-specific features based on their auditory importance, enabling precise detection across varied instrument types and combinations. Enhanced by track classification and permutation augmentation, our model substantially outperforms existing SVM and CRNN models, showing robustness on unseen instruments and out-of-domain testing. We believe our exploration provides valuable insights for future research on audio content analysis in multitrack music settings.

Problem

• Detects lead instrument in multitrack music audio
• Overcomes limitations of coarse classification and generalization
• Uses self-supervised learning and attention-based classifier

Innovation

• Self-supervised learning model integration
• Track-wise frame-level attention mechanism
• Track classification and permutation augmentation