InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring

2026-07-16

AI Summary

This work addresses the limitations of existing in-vehicle affective computing datasets, which are predominantly constrained to visual modalities and lack linguistic and interactive cues. To bridge this gap, we introduce InCarEmo, a novel multimodal in-car dataset that uniquely integrates RGB and infrared video, in-cabin audio, and bilingual (Chinese–English) conversational transcripts, captured across diverse lighting conditions and real-world driving scenarios. The dataset supports three core tasks: emotion recognition, fatigue detection, and distraction monitoring. We establish cross-lingual benchmarks and systematically evaluate model robustness under modality dropout and realistic noise conditions. Experimental results demonstrate that multimodal fusion substantially enhances performance, while also revealing persistent challenges in low-light settings and noisy environments, thereby providing a comprehensive and practical benchmark for in-vehicle affective understanding.

Abstract

Understanding driver emotion and state is critical for the next generation of intelligent in-cabin systems that ensure safety and enhance human-vehicle interaction. However, existing public datasets for in-cabin affective computing are largely limited to visual modalities and rarely include conversational information, making it difficult to capture the linguistic and interactive cues underlying driver emotion. To address these gaps, we introduce InCarEmo, a multimodal dataset for in-cabin emotion recognition and driver state monitoring. InCarEmo integrates RGB and infrared video, in-cabin audio, and dialogue text collected from scripted in-cabin scenarios designed to simulate realistic driver behaviors, covering diverse lighting conditions and driving contexts. The dataset supports three primary tasks: 1) multimodal emotion recognition, 2) fatigue detection, and 3) distraction monitoring. In addition to the original Chinese data, we construct an auxiliary English benchmark to support preliminary cross-lingual evaluation. We provide a unified benchmark with extensive baseline results across unimodal and multimodal methods, including analyses under modality-missing and noise conditions. Experimental results demonstrate the benefits of multimodal fusion and reveal remaining challenges under real-world noise and low-light conditions. By releasing InCarEmo, we aim to establish a comprehensive foundation for robust, interpretable, and human-centric in-cabin affective understanding, promoting safer and more empathetic driver-vehicle interaction.

Problem

• in-cabin emotion recognition
• driver state monitoring
• multimodal dataset
• affective computing
• human-vehicle interaction

Innovation

• multimodal dataset
• in-cabin emotion recognition
• driver state monitoring
• cross-lingual benchmark
• dialogue-augmented sensing