Beyond Overt Reactions: Analyzing Subtle User Emotional Response to Unexpected In-Vehicle System Behavior

2026-08-15

AI Summary

This study addresses the challenges of recognizing subtle user emotions and data scarcity associated with unexpected autonomous driving behaviors. Leveraging a driving simulator, we collected video, audio, and heart rate signals to construct the first multimodal dataset capturing subtle in-vehicle emotional responses during unexpected events. By integrating affective computing with behavioral analysis, this work elucidates the characteristics of users' subtle emotional reactions to sudden incidents, thereby bridging a critical data gap in the field. The findings validate the necessity for vehicles to perceive and adapt to user states, providing essential empirical evidence and theoretical foundations for enhancing the safety and adaptive capabilities of human-machine interaction in autonomous driving systems.

Abstract

Modern vehicles, with advanced AI voice and autonomous navigation features, extend beyond traditional driving but, like any autonomous system, can potentially make mistakes or behave in ways unexpected by users. Although providing real-time explanations can alleviate some confusion, constant information can overwhelm users and potentially cause unnecessary distractions. Some situations may require explanations or corrective vehicle behavior, and thus, recognizing user response to unexpected vehicle behavior is critical. To investigate such user responses, our study focused on collecting and analyzing user behavioral responses to unexpected events while interacting with a fully autonomous vehicle in a driving simulator. We also aimed to address the lack of datasets capturing subtle user responses (facial, spoken language, physiological signals) to in-vehicle events, as existing datasets primarily focus on strong emotional signals in conventional human-driven cars and user response to external road and traffic conditions. Users were exposed to stimuli designed to induce surprise, confusion, and frustration while performing a secondary task on a tablet and interacting with the vehicle through voice commands and in-vehicle displays. We collected a multi-modal dataset with video, audio, and heart rate data and gained insights into subtle user responses that underscored the need for further investigation of nuanced user behaviors. These observations highlight the importance of designing vehicles that recognize and adapt to occupants' behavior, potentially improving their experience.

Problem

• Unexpected In-Vehicle System Behavior
• Subtle User Emotional Response
• Autonomous Vehicles
• Multi-modal Dataset
• Human-Vehicle Interaction

Innovation

• Subtle Emotional Response
• Multi-modal Dataset
• Unexpected Vehicle Behavior
• Autonomous Driving
• Physiological Signals