Designing LLM-simulated Immersive Spaces to Enhance Autistic Children's Social Affordances Understanding

2025-02-05

AI Summary

Children with autism spectrum disorder (ASD) struggle to recognize social cues in traffic contexts, posing safety risks. To address this, we propose AIroad: an LLM-driven immersive projection environment featuring interpretable and controllable virtual driver agents that simulate multimodal social signals (e.g., speech, facial expressions, gestures) to support training in social affordance perception. We introduce 17 novel design principles for LLM-integrated immersive environments tailored to children with ASD and implement a closed-loop learning framework linking perceived social intent to appropriate behavioral responses. A user study (n=14) demonstrates statistically significant improvement in social affordance understanding (p<0.01) and high engagement; caregiver feedback indicates strong usability and acceptance. This work represents the first integration of large language models into embodied social reasoning training, establishing a new paradigm for AI-mediated interventions targeting neurodiverse populations.

Technology Category

• Humans and AI: AI for Accessibility
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

One of the key challenges faced by autistic children is understanding social affordances in complex environments, which further impacts their ability to respond appropriately to social signals. In traffic scenarios, this impairment can even lead to safety concerns. In this paper, we introduce an LLM-simulated immersive projection environment designed to improve this ability in autistic children while ensuring their safety. We first propose 17 design considerations across four major categories, derived from a comprehensive review of previous research. Next, we developed a system called AIroad, which leverages LLMs to simulate drivers with varying social intents, expressed through explicit multimodal social signals. AIroad helps autistic children bridge the gap in recognizing the intentions behind behaviors and learning appropriate responses through various stimuli. A user study involving 14 participants demonstrated that this technology effectively engages autistic children and leads to significant improvements in their comprehension of social affordances in traffic scenarios. Additionally, parents reported high perceived usability of the system. These findings highlight the potential of combining LLM technology with immersive environments for the functional rehabilitation of autistic children in the future.

Problem

• Enhancing social affordances understanding in autistic children
• Improving safety in traffic scenarios for autistic children
• Developing LLM-simulated immersive environments for autism rehabilitation

Innovation

• LLM-simulated immersive projection environment
• AIroad system with multimodal social signals
• Design considerations for autism rehabilitation