UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions

2025-11-05

AI Summary

Existing open-source audio-visual generation methods exhibit weak cross-modal modeling capabilities, resulting in lip-audio asynchrony and semantic inconsistency. To address this, we propose UniAVGen—the first unified audio-visual generation framework based on a dual-branch diffusion Transformer. Its core contributions are: (1) an asymmetric cross-modal interaction mechanism enabling spatiotemporal audio-video alignment; (2) a face-aware modulation module that enhances dynamic lip motion modeling; and (3) a modality-aware classifier-free guidance strategy improving multi-task generalization. The framework integrates bidirectional temporal alignment cross-attention with a dynamic region-weighting mechanism. Extensive experiments demonstrate that, trained on only 1.3M samples, UniAVGen surpasses large-scale baseline models in lip-audio synchronization accuracy, timbre fidelity, and emotional coherence—achieving state-of-the-art performance with significantly reduced data requirements.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Due to the lack of effective cross-modal modeling, existing open-source audio-video generation methods often exhibit compromised lip synchronization and insufficient semantic consistency. To mitigate these drawbacks, we propose UniAVGen, a unified framework for joint audio and video generation. UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which enables bidirectional, temporally aligned cross-attention, thus ensuring precise spatiotemporal synchronization and semantic consistency. Furthermore, this cross-modal interaction is augmented by a Face-Aware Modulation module, which dynamically prioritizes salient regions in the interaction process. To enhance generative fidelity during inference, we additionally introduce Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint synthesis design enables seamless unification of pivotal audio-video tasks within a single model, such as joint audio-video generation and continuation, video-to-audio dubbing, and audio-driven video synthesis. Comprehensive experiments validate that, with far fewer training samples (1.3M vs. 30.1M), UniAVGen delivers overall advantages in audio-video synchronization, timbre consistency, and emotion consistency.

Problem

• Improving lip synchronization in audio-video generation
• Enhancing semantic consistency across audio and video modalities
• Unifying multiple audio-video tasks within a single model

Innovation

• Dual-branch joint synthesis with Diffusion Transformers
• Asymmetric Cross-Modal Interaction for spatiotemporal synchronization
• Modality-Aware Classifier-Free Guidance for cross-modal correlation