MeloKids: Multisensory VR System to Enhance Speech and Motor Coordination in Children with Hearing Loss

2025-07-24

AI Summary

To address language and motor developmental delays in children with hearing impairment, this study developed a virtual reality (VR)-based multisensory training system integrating auditory, visual, and tactile feedback, centered on pitch-matching tasks to concurrently enhance speech production and motor coordination. Innovatively, it represents the first integration of VR with functional near-infrared spectroscopy (fNIRS) neurofeedback, enabling real-time monitoring and closed-loop neuromodulation of prefrontal and motor cortical activation during rehabilitation. By quantifying neural responses—specifically oxyhemoglobin (HbO) concentration changes—and behavioral metrics—including movement accuracy and reaction time—the intervention demonstrated significant increases in cortical activation, heightened cognitive engagement, and improved motor control in hearing-impaired children. These findings establish a novel paradigm for personalized, immersive neurorehabilitation grounded in objective neurobehavioral biomarkers.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Children with hearing impairments face ongoing challenges in language and motor development. This study explores how multi-sensory feedback technology based on virtual reality (VR), integrating auditory, visual, and tactile stimuli, can enhance rehabilitation outcomes. Using functional near-infrared spectroscopy (fNIRS) technology, we assessed cortical activation patterns in children during pitch-matching tasks across different interaction modes. Our findings aim to provide evidence for designing personalized, interactive rehabilitation systems that enhance cognitive engagement and motor control in children with hearing impairments.

Problem

• Enhance speech and motor coordination in hearing-impaired children
• Explore multi-sensory VR for rehabilitation outcomes
• Assess cortical activation during pitch-matching tasks

Innovation

• VR integrates auditory, visual, tactile stimuli
• fNIRS assesses cortical activation patterns
• Personalized interactive rehabilitation systems enhance engagement