UTI-LLM: A Personalized Articulatory-Speech Therapy Assistance System Based on Multimodal Large Language Model

2025-09-16

AI Summary

Current speech therapy systems lack real-time articulatory feedback, while multimodal large language models (MLLMs) face bottlenecks in speech rehabilitation—including insufficient acquisition and fusion of articulatory information, coarse-grained tongue motion trajectory analysis, and scarcity of domain-specific data. To address these challenges, we introduce the first high-quality, ultrasound tongue imaging–speech paired dataset specifically designed for speech therapy. We propose a spatiotemporal fusion training strategy integrating ultrasound video and acoustic signals, and develop a novel MLLM that jointly processes ultrasound imagery, raw speech waveforms, and textual dialogue. Leveraging cross-modal alignment and data-driven fine-tuning, our model enables fine-grained spatiotemporal modeling of tongue dynamics and generates clinically interpretable, real-time articulatory feedback. Evaluated in authentic clinical settings, the system significantly improves phoneme error detection accuracy and feedback latency/quality, thereby enhancing rehabilitation precision, interactivity, and scalability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Speech therapy plays a critical role in training speech disorders caused by neurological impairments such as stroke. However, traditional manual and computer-assisted systems are limited in real-time accessibility and articulatory motion feedback, constraining their practical utility. Recent advances in multimodal large language models (MLLMs) have demonstrated significant potential in healthcare, particularly through their ability to integrate multimodal data for adaptive assessment and therapeutic feedback. Nevertheless, challenges including insufficient acquisition and fusion of articulatory information, inadequate parsing of articulatory organ motion trajectories, and the scarcity of high-quality domain-specific datasets hinder the application of MLLMs in speech therapy. To address these limitations, we propose an MLLM-based speech rehabilitation assistance system that synergistically leverages ultrasound tongue imaging and speech signals to deliver precise, interactive articulatory feedback. We construct a high-quality domain-specific dataset comprising UTI-speech dialogue pairs. This dataset facilitates fine-tuning to enhance the model's clinical adaptability. Building on this dataset, our methods achieves spatiotemporal fusion training strategy of ultrasound videos and speech signals, enabling fine-grained articulatory impairment analysis and ultimately generating actionable feedback.

Problem

• Addresses insufficient articulatory data acquisition and fusion in speech therapy
• Overcomes inadequate parsing of articulatory organ motion trajectories
• Solves scarcity of high-quality domain-specific datasets for MLLMs

Innovation

• Multimodal fusion of ultrasound and speech signals
• Spatiotemporal training strategy for fine-grained analysis
• Domain-specific dataset for clinical adaptability enhancement