Giving Meaning to Movements: Challenges and Opportunities in Expanding Communication by Pairing Unaided AAC with Speech Generated Messages

2026-02-25

AI Summary

This study addresses the tension between naturalness and intelligibility in augmentative and alternative communication (AAC) for individuals with communication impairments—where unaided AAC tends to be natural yet poorly understood, while aided AAC is highly intelligible but often lacks expressiveness. Through an 18-month participatory design process, the authors developed AllyAAC, a hybrid AAC system integrating a wrist-worn inertial measurement unit (IMU) with a mobile application. They constructed the first multimodal dataset comprising over 600,000 atypical gestures and introduced a personalized gesture recognition approach based on a Transformer-based large model with a differentiated pretraining strategy. Field evaluations with 14 users demonstrated the system’s effectiveness, yielding adaptive design principles and a reference implementation for hybrid AAC systems.

Technology Category

• Humans and AI: AI for Accessibility
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Augmentative and Alternative Communication (AAC) technologies are categorized into two forms: aided AAC, which uses external devices like speech-generating systems to produce standardized output, and unaided AAC, which relies on body-based gestures for natural expression but requires shared understanding. We investigate how to combine these approaches to harness the speed and naturalness of unaided AAC while maintaining the intelligibility of aided AAC, a largely unexplored area for individuals with communication and motor impairments. Through 18 months of participatory design with AAC users, we identified key challenges and opportunities and developed AllyAAC, a wearable system with a wrist-worn IMU paired with a smartphone app. We evaluated AllyAAC in a field study with 14 participants and produced a dataset containing over 600,000 multimodal data points featuring atypical gestures--the first of its kind. Our findings reveal challenges in recognizing personalized, idiosyncratic gestures and demonstrate how to address them using Transformer-based large machine learning (ML) models with different pretraining strategies. In sum, we contribute design principles and a reference implementation for adaptive, personalized systems combining aided and unaided AAC.

Problem

• Augmentative and Alternative Communication
• unaided AAC
• aided AAC
• gesture recognition
• communication impairment

Innovation

• unaided AAC
• wearable IMU
• personalized gesture recognition
• Transformer-based ML
• multimodal dataset