Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling

2026-04-26

AI Summary

Existing joint audio-visual generation methods couple modalities throughout the denoising process, entangling high-level semantics with low-level details and thereby compromising the efficiency and quality of talking-head synthesis. To address this, this work proposes an autoregressive diffusion framework that decouples semantic modeling from rendering: a shared backbone captures high-level cross-modal semantics, while modality-specific lightweight diffusion Transformer decoders independently refine low-level audio and visual details. By unifying audio and video processing at the patch-level token representation, the proposed approach significantly outperforms dual-branch baselines on standard talking-head benchmarks, achieving state-of-the-art performance in lip-sync accuracy, video quality, and audio fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Joint audio-video generation models have shown that unified generation yields stronger cross-modal coherence than cascaded approaches. However, existing models couple modalities throughout denoising via pervasive attention, treating high-level semantics and low-level details in a fully entangled manner. This is suboptimal for talking head synthesis: while audio and facial motion are semantically correlated, their low-level realizations (acoustic signals and visual textures) follow distinct rendering processes. Enforcing joint modeling across all levels causes unnecessary entanglement and reduces efficiency. We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone, while low-level refinement uses modality-specific decoders. A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space. Two lightweight diffusion transformer heads decode the hidden states into frame-level audio and video latents. Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality, achieving stronger cross-modal consistency than cascaded pipelines.

Problem

• talking head synthesis
• audio-video generation
• cross-modal coherence
• modality entanglement
• diffusion modeling

Innovation

• autoregressive diffusion
• cross-modal disentanglement
• talking head synthesis
• modality-specific decoding
• unified token space