MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning

2026-01-04

arXiv.org

AI Summary

Existing unified audio-visual generation models face limitations in fine-grained acoustic control, speaker identity preservation, temporal alignment, and zero-shot voice cloning. To address these challenges, this work proposes MM-Sonate, a flow-matching-based multimodal generative framework that achieves strict audio-visual synchronization through joint instruction-phoneme encoding and disentangles speaker identity from linguistic content via a timbre injection mechanism. Furthermore, a negative conditional generation strategy grounded in a noise prior is introduced to enhance acoustic fidelity. Experimental results demonstrate that MM-Sonate significantly outperforms current methods in lip-sync accuracy, speech intelligibility, and zero-shot cloning fidelity, establishing a new state of the art on joint audio-visual generation benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Joint audio-video generation aims to synthesize synchronized multisensory content, yet current unified models struggle with fine-grained acoustic control, particularly for identity-preserving speech. Existing approaches either suffer from temporal misalignment due to cascaded generation or lack the capability to perform zero-shot voice cloning within a joint synthesis framework. In this work, we present MM-Sonate, a multimodal flow-matching framework that unifies controllable audio-video joint generation with zero-shot voice cloning capabilities. Unlike prior works that rely on coarse semantic descriptions, MM-Sonate utilizes a unified instruction-phoneme input to enforce strict linguistic and temporal alignment. To enable zero-shot voice cloning, we introduce a timbre injection mechanism that effectively decouples speaker identity from linguistic content. Furthermore, addressing the limitations of standard classifier-free guidance in multimodal settings, we propose a noise-based negative conditioning strategy that utilizes natural noise priors to significantly enhance acoustic fidelity. Empirical evaluations demonstrate that MM-Sonate establishes new state-of-the-art performance in joint generation benchmarks, significantly outperforming baselines in lip synchronization and speech intelligibility, while achieving voice cloning fidelity comparable to specialized Text-to-Speech systems.

Problem

• joint audio-video generation
• zero-shot voice cloning
• temporal alignment
• speaker identity preservation
• multimodal synthesis

Innovation

• multimodal generation
• zero-shot voice cloning
• flow matching
• timbre injection
• noise-based conditioning