Frequency-Aware Self-Supervised Music Representation Learning

2026-06-24

AI Summary

This work addresses a key limitation in existing self-supervised music representation learning methods, which treat audio as a one-dimensional sequence and thereby overlook the rich two-dimensional structure of time–frequency spectrograms, hindering effective modeling of music’s intrinsic characteristics. To overcome this, the authors propose PupuJEPA, the first visual joint-embedding predictive architecture operating directly on two-dimensional spectrograms. It leverages masked spectrogram patches and predicts their latent embeddings in a self-supervised manner, incorporating music-specific priors into its model design, training strategy, and inference paradigm. An attention mechanism is further introduced to interpret musical semantic structures. Evaluated on the MARBLE benchmark, PupuJEPA achieves substantially superior linear probing performance compared to state-of-the-art one-dimensional sequence-based self-supervised models, and attention visualizations confirm its ability to capture meaningful musical structures.

Abstract

Self-supervised learning (SSL) has emerged as an essential paradigm for music information retrieval (MIR). While current SSL models achieve state-of-the-art performance across various MIR tasks, they typically treat audio as 1D sequences, either operating on time-domain waveforms or on flattened time-frequency-domain spectrograms. This discards the rich spatial and structural information in time-frequency representations and overlooks a fundamental intuition in music production. In particular, music is naturally represented as time-frequency grids in MIDI-based workflows, a structure that tightly corresponds to 2D spectrograms and inherently makes many MIR tasks trivial. Motivated by this intuition, we propose PupuJEPA, a visual Joint-Embedding Predictive Architecture (JEPA) that is trained directly on 2D spectrograms. Instead of applying masked language modeling (MLM) to 1D sequences, PupuJEPA learns robust representations by predicting the latent embeddings of masked 2D spectrogram patches from unmasked contexts. To optimally adapt such a visual framework to music signals, we also apply domain-specific modifications to model architecture, training scheme, and inference paradigm, with comprehensive ablation studies showing their effectiveness. Evaluations on the MARBLE benchmark show that PupuJEPA outperforms the 1D sequence-based SSL models across multiple MIR tasks in linear probing. Additionally, case studies of the attention maps also confirm that PupuJEPA captures musically meaningful patterns within the 2D time-frequency domain. Codes and checkpoints are available at: https://www.yichenggu.com/PupuJEPA/.

Problem

• self-supervised learning
• music information retrieval
• time-frequency representation
• 2D spectrogram
• representation learning

Innovation

• self-supervised learning
• time-frequency representation
• 2D spectrogram
• Joint-Embedding Predictive Architecture
• music information retrieval