A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI

2025-03-15

AI Summary

This study addresses the challenge of synthesizing real-time or cine magnetic resonance imaging (RT-/cine-MRI) videos from speech signals to visualize dynamic articulatory tract anatomy during phonation, supporting clinical assessment and personalized rehabilitation. We propose the first spatiotemporal diffusion framework integrating anatomical structure guidance and temporal modeling: leveraging rigorously time-aligned speech–MRI preprocessing, we adapt the Stable Diffusion architecture by incorporating an anatomical structure constraint loss and a temporal attention mechanism. The method significantly improves cross-subject generalization and real-time generation capability for unseen utterances. Validated on data from healthy participants and tongue cancer patients, the synthesized videos exhibit anatomically plausible structures and high motion fidelity. Quantitative metrics and expert clinical evaluation jointly confirm the method’s clinical utility and visualization accuracy.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Speech
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Understanding the relationship between vocal tract motion during speech and the resulting acoustic signal is crucial for aided clinical assessment and developing personalized treatment and rehabilitation strategies. Toward this goal, we introduce an audio-to-video generation framework for creating Real Time/cine-Magnetic Resonance Imaging (RT-/cine-MRI) visuals of the vocal tract from speech signals. Our framework first preprocesses RT-/cine-MRI sequences and speech samples to achieve temporal alignment, ensuring synchronization between visual and audio data. We then employ a modified stable diffusion model, integrating structural and temporal blocks, to effectively capture movement characteristics and temporal dynamics in the synchronized data. This process enables the generation of MRI sequences from new speech inputs, improving the conversion of audio into visual data. We evaluated our framework on healthy controls and tongue cancer patients by analyzing and comparing the vocal tract movements in synthesized videos. Our framework demonstrated adaptability to new speech inputs and effective generalization. In addition, positive human evaluations confirmed its effectiveness, with realistic and accurate visualizations, suggesting its potential for outpatient therapy and personalized simulation of vocal tract visualizations.

Problem

• Understand vocal tract motion and acoustic signal relationship.
• Generate RT-/cine-MRI visuals from speech signals.
• Improve audio-to-video conversion for clinical applications.

Innovation

• Audio-to-video generation for vocal tract MRI
• Modified stable diffusion model for synchronization
• Realistic vocal tract visualization from speech