Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

2026-08-16

AI Summary

This study addresses the lack of speaker controllability in text-to-audio-video generation models by proposing an efficient voice cloning method. By injecting zero-initialized linear layers and reference signals into the audio backbone, combined with global speaker embedding modulation and short-term fine-tuning, the original model is transformed into a controllable framework supporting independent audio-path inference. Remarkably, this efficient adaptation is achieved using only a single zero-initialized layer. Experiments on a 30-speaker benchmark demonstrate that the proposed method significantly outperforms five strong baselines in Speaker Embedding Cosine Similarity (SECS). Furthermore, it achieves an approximately 30-fold speedup in independent audio inference, effectively balancing generation controllability with computational efficiency.

Abstract

Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.

Problem

• Text-to-Audio-Video
• Voice Cloning
• Speaker Control
• T2AV Generation

Innovation

• Voice Cloning
• Zero-Initialized Layer
• Text-to-Audio-Video
• Dual Reference Injection
• Inference Acceleration