OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL

2026-06-29

AI Summary

Self-supervised speech representation learning struggles to simultaneously satisfy diverse objectives such as waveform generation, speaker identification, and semantic understanding. To address this challenge, this work proposes OLIVE, a unified framework that jointly models view-augmented masked latent prediction and end-to-end waveform reconstruction. By co-optimizing analysis and synthesis objectives, OLIVE preserves fine-grained acoustic details in early encoder layers while encouraging high-level representations to be task-invariant. Within a single architecture, the method achieves concurrent improvements in generative quality and robustness on downstream tasks, significantly outperforming existing approaches in speech synthesis and speaker-related benchmarks, maintaining competitive performance in semantic and recognition tasks, and enabling higher-fidelity waveform reconstruction.

Abstract

We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE combines view-augmented masked latent prediction with waveform reconstruction under a unified objective. Reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations toward invariance for robust downstream performance. We show that these objectives enable representations that support a broad range of tasks. In particular, OLIVE improves results on generation and speaker tasks, maintains competitive performance on recognition and semantic tasks, and improves waveform reconstruction.

Problem

• self-supervised learning
• speech representation
• waveform reconstruction
• masked latent prediction
• multi-task learning

Innovation

• self-supervised learning
• masked latent prediction
• waveform reconstruction
• view augmentation
• speech representation learning