AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation

2025-04-29

AI Summary

This work addresses core challenges in multimodal speech generation—low speech intelligibility, audio-visual desynchronization, poor naturalness, and weak speaker similarity—by proposing the first text-video-reference-audio tri-modal co-driven high-fidelity speech synthesis framework. Methodologically, we design a multimodal-aligned Diffusion Transformer (DiT) architecture, incorporating a modality-adaptive multimodal classifier-free guidance mechanism and a fine-grained audio-visual temporal synchronization modeling module. Key technical contributions include: (i) a cross-modal feature alignment strategy; (ii) multimodal conditional encoding; and (iii) dynamic weight fusion for guided generation. Extensive evaluations demonstrate significant improvements over state-of-the-art methods across multiple benchmarks, with substantial gains in speech quality (MOS), synchronization accuracy (SyncScore), and speaker similarity (SIM). The framework further generalizes effectively to video-to-speech and vision-forced alignment tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

In this paper, we address the task of multimodal-to-speech generation, which aims to synthesize high-quality speech from multiple input modalities: text, video, and reference audio. This task has gained increasing attention due to its wide range of applications, such as film production, dubbing, and virtual avatars. Despite recent progress, existing methods still suffer from limitations in speech intelligibility, audio-video synchronization, speech naturalness, and voice similarity to the reference speaker. To address these challenges, we propose AlignDiT, a multimodal Aligned Diffusion Transformer that generates accurate, synchronized, and natural-sounding speech from aligned multimodal inputs. Built upon the in-context learning capability of the DiT architecture, AlignDiT explores three effective strategies to align multimodal representations. Furthermore, we introduce a novel multimodal classifier-free guidance mechanism that allows the model to adaptively balance information from each modality during speech synthesis. Extensive experiments demonstrate that AlignDiT significantly outperforms existing methods across multiple benchmarks in terms of quality, synchronization, and speaker similarity. Moreover, AlignDiT exhibits strong generalization capability across various multimodal tasks, such as video-to-speech synthesis and visual forced alignment, consistently achieving state-of-the-art performance. The demo page is available at https://mm.kaist.ac.kr/projects/AlignDiT .

Problem

• Generate high-quality speech from text, video, and audio inputs
• Improve speech intelligibility, synchronization, and naturalness
• Enhance voice similarity to reference speaker in multimodal synthesis

Innovation

• Multimodal Aligned Diffusion Transformer for synchronized speech
• Novel classifier-free guidance balances multimodal inputs
• DiT architecture enhances in-context learning capability