Making Separation-First Multi-Stream Audio Watermarking Feasible via Joint Training

2026-03-17

AI Summary

This work addresses the challenge of accurately recovering individual watermarks from separated audio tracks in multi-track mixing and separation scenarios, where conventional watermarking methods often fail. To this end, we propose the first “separation-first” end-to-end joint training framework that simultaneously optimizes an audio separator and a multi-stream watermarking system. By integrating shared-structure multi-key watermark embedding, off-the-shelf separation model adaptation, and a joint training strategy, our approach enables watermark embedding to be robust to separation-induced distortions while encouraging the separator to preserve watermark-critical features. Experimental results demonstrate significant improvements in post-separation watermark bit recovery rates on both speech-plus-music and vocal-plus-accompaniment mixtures, all while maintaining high perceptual audio quality.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Motion & Tracking

Application Category

• Security and Privacy: Tracking, profiling, and countermeasures against them
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Modern audio is created by mixing stems from different sources, raising the question: can we independently watermark each stem and recover all watermarks after separation? We study a separation-first, multi-stream watermarking framework-embedding distinct information into stems using unique keys but a shared structure, mixing, separating, and decoding from each output. A naive pipeline (robust watermarking + off-the-shelf separation) yields poor bit recovery, showing robustness to generic distortions does not ensure robustness to separation artifacts. To enable this, we jointly train the watermark system and the separator in an end-to-end manner, encouraging the separator to preserve watermark cues while adapting embedding to separation-specific distortions. Experiments on speech+music and vocal+accompaniment mixtures show substantial gains in post-separation recovery while maintaining perceptual quality.

Problem

• audio watermarking
• source separation
• multi-stream
• watermark recovery
• separation artifacts

Innovation

• audio watermarking
• source separation
• joint training
• multi-stream
• separation-first