BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations

2026-06-29

AI Summary

This work proposes a two-stage pretraining strategy to enhance the cross-domain and cross-task transferability of self-supervised audio representations. In the first stage, a Vision Transformer (ViT) serves as a context encoder to process unmasked spectrogram regions; in the second stage, a lightweight predictor reconstructs randomly projected and discretized masked targets. After pretraining, the predictor is discarded, and only the ViT encoder is retained for downstream tasks. By decoupling masked modeling into distinct context learning and prediction phases—and replacing the conventional Conformer architecture with ViT—the approach achieves substantially improved performance balance across diverse domains. Evaluated on the X-ARES and XARES-LLM benchmarks, the model demonstrates superior overall transfer performance compared to single-stage baselines while maintaining identical inference computational costs.

Abstract

Self-supervised learning enables audio representations that transfer across domains and tasks. We present BEST-RQ-2, an evolution of BEST-RQ that retains frozen randomprojection-based discrete targets while introducing a two-step contextualize-then-predict pretraining scheme. A ViT context encoder processes only the unmasked spectrogram regions, and a lightweight predictor infers targets for the masked regions; the predictor is discarded after pretraining. Replacing the original Conformer encoder with a ViT shifts performance across domains, slightly reducing speech performance while improving music and environmental sounds, with comparable average scores. The main improvement comes from decomposing masked prediction into separate contextualization and prediction stages. On the X-ARES and XARES-LLM benchmarks, BEST-RQ-2 consistently outperforms one-stage baselines in overall transfer while keeping inference compute unchanged. Code and model checkpoints are publicly available.

Problem

• self-supervised learning
• audio representations
• masked prediction
• cross-domain transfer
• pretraining

Innovation

• two-step pretraining
• contextualize-then-predict
• self-supervised audio representation
• masked prediction decomposition
• Vision Transformer (ViT)