MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning

2026-06-23

AI Summary

This work addresses the limitations of existing audio-visual self-supervised learning methods, which rely on modality-specific encoders and complex objective functions that hinder effective cross-modal synergy. The authors propose the first Joint Embedding Predictive Architecture (JEPA) for audio-visual representation learning, featuring a modality-agnostic unified encoder and a single predictive objective that jointly models intra- and inter-modal relationships to enable complementary information exchange across modalities. With a frozen ViT-g backbone, the method surpasses the previous best frozen baseline by 6.8 mAP on AudioSet-20K and outperforms fully fine-tuned models on ESC-50 and FSD50K. Notably, it achieves competitive performance on video tasks using only one-tenth of the video data, demonstrating substantially improved representation efficiency and generalization capability.

Abstract

Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning. Since audio and visual streams naturally co-occur in video data, extending this success to jointly learn from both modalities is a natural next step, yet it remains challenging. Existing audio-visual self-supervised methods rely on modality-specific encoders and complex combinations of contrastive or reconstruction objectives, limiting cross-modal synergy and scalability. Joint Embedding Predictive Architectures (JEPAs) offer a simple, modality-agnostic alternative, but have to date been applied primarily to individual modalities. We introduce MJEPA, a joint-embedding predictive architecture for audio-visual learning that uses a single, unified encoder for both modalities. Our approach uses only a single predictive objective, applied both within and across modalities. We show that cross-modal prediction is critical: without it, a shared encoder degrades below unimodal baselines; with it, each modality's representation benefits from the other. Our frozen ViT-g model outperforms the best prior frozen baseline by over 6.8 mAP on AudioSet-20K, surpasses fully finetuned models on ESC-50 and FSD50K, and is competitive on video benchmarks despite using 10x less video data.

Problem

• audio-visual learning
• self-supervised learning
• joint embedding
• cross-modal representation
• scalability

Innovation

• joint-embedding
• audio-visual learning
• self-supervised learning
• cross-modal prediction
• unified encoder