Hierarchical Codec Diffusion for Video-to-Speech Generation

2026-04-17

AI Summary

This work addresses the limitation of existing video-to-speech generation methods, which often overlook the hierarchical structure of speech and struggle to achieve multi-granularity audiovisual alignment. The authors propose HiCoDiT, a novel framework that introduces, for the first time in this task, a discrete hierarchical speech codec based on residual vector quantization (RVQ), integrated with a hierarchical diffusion Transformer to separately model coarse-grained speaker semantics and fine-grained prosodic features. To effectively fuse global timbre and local prosody, they further design a dual-scale adaptive instance layer normalization mechanism. Experimental results demonstrate that HiCoDiT significantly outperforms current baselines in both speech fidelity and expressiveness, validating the efficacy of hierarchical discrete modeling for video-driven speech synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Speech
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalization and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT.

Problem

• Video-to-Speech
• hierarchical speech representation
• audio-visual alignment
• prosody modeling
• speaker-aware synthesis

Innovation

• Hierarchical Codec Diffusion
• Video-to-Speech Generation
• Residual Vector Quantization
• Dual-scale Adaptive Normalization
• Discrete Speech Tokens