ImageTalk: Designing a Multimodal AAC Text Generation System Driven by Image Recognition and Natural Language Generation

2025-12-10

AI Summary

Patients with motor neuron disease (MND) frequently rely on augmentative and alternative communication (AAC) systems due to progressive speech and motor impairments; however, conventional symbol-based AAC systems suffer from limited vocabulary coverage and low text-input efficiency. This paper proposes a multimodal AAC text generation system specifically designed for MND users, integrating image recognition and natural language generation (NLG), and developed via a co-design paradigm involving both proxy and end users. We introduce three AI-augmented AAC design principles and a four-tier user needs model. The system’s efficacy is rigorously validated through iterative, real-user closed-loop evaluation. Experimental results demonstrate a 95.6% reduction in keystrokes, stable response latency, and high user satisfaction. This work delivers a clinically deployable, highly usable multimodal intelligent text generation framework for AAC, advancing practical assistive technology for neurodegenerative conditions.

Technology Category

• Humans and AI: AI for Accessibility
• Machine Learning: Multimodal Learning
• Multiagent Systems: Agent Communication

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

People living with Motor Neuron Disease (plwMND) frequently encounter speech and motor impairments that necessitate a reliance on augmentative and alternative communication (AAC) systems. This paper tackles the main challenge that traditional symbol-based AAC systems offer a limited vocabulary, while text entry solutions tend to exhibit low communication rates. To help plwMND articulate their needs about the system efficiently and effectively, we iteratively design and develop a novel multimodal text generation system called ImageTalk through a tailored proxy-user-based and an end-user-based design phase. The system demonstrates pronounced keystroke savings of 95.6%, coupled with consistent performance and high user satisfaction. We distill three design guidelines for AI-assisted text generation systems design and outline four user requirement levels tailored for AAC purposes, guiding future research in this field.

Problem

• Designing a multimodal AAC system for motor neuron disease patients
• Overcoming limited vocabulary and slow text entry in traditional AAC
• Enabling efficient communication through image recognition and language generation

Innovation

• Image recognition drives multimodal AAC text generation
• Natural language generation enhances communication efficiency
• Proxy-user and end-user design phases ensure tailored system