Voice-Controlled Scratch for Children with (Motor) Disabilities

2026-03-30

AI Summary

This study addresses the challenge faced by children with motor impairments in using Scratch’s mouse-based drag-and-drop interface, noting that existing assistive technologies predominantly target visual impairments and offer limited support for physical interaction barriers. To bridge this gap, the authors present the first voice-based programming system for Scratch tailored to this population, leveraging a multimodal voice user interface that integrates digital overlays and spoken labels to entirely eliminate the need for physical input. The system introduces an innovative multi-stage speech recognition pipeline combining regular expressions, phoneme matching, and custom grammars, augmented by numeric stacking and voice navigation mechanisms to robustly handle child speech patterns and classroom noise. Experimental results demonstrate an overall command recognition accuracy of 82.8%, with simple commands achieving 96.9% accuracy—substantially outperforming a baseline general-purpose speech recognition system, which attained only 46.4%.

Abstract

Block-based programming environments like Scratch have become widely adopted in Computer Science Education, but the mouse-based drag-and-drop interface can challenge users with disabilities. While prior work has provided solutions supporting children with visual impairment, these solutions tend to focus on making content perceivable and do not address the physical interaction barriers faced by users with motor disabilities. To bridge this gap, we introduce MeowCrophone, an approach that uses voice control to allow editing code in Scratch. MeowCrophone supports clicking elements, placing blocks, and navigating the workspace via a multi-modal voice user interface that uses numerical overlays and label reading to bypass physical input entirely. As imperfect speech recognition is common in classrooms and for children with dysarthria, MeowCrophone employs a multi-stage matching pipeline using regular expressions, phonetic matching, and a custom grammar. Evaluation shows that while free speech recognition systems achieved a baseline success rate of only 46.4%, MeowCrophone's pipeline improved results to 82.8% overall, with simple commands reaching 96.9% accuracy. This demonstrates that robust voice control can make Scratch accessible to users for whom visual aids are insufficient.

Problem

• motor disabilities
• block-based programming
• physical interaction barriers
• voice control
• accessibility

Innovation

• voice-controlled programming
• accessibility
• Scratch
• speech recognition pipeline
• motor disabilities