DIPSER: A Dataset for In-Person Student1 Engagement Recognition in the Wild

2025-02-27

AI Summary

This study addresses the challenge of accurately recognizing students’ attention and affective states in authentic classroom settings. To this end, we introduce the first multimodal dataset for student attentiveness recognition specifically designed for in-person instruction. Our methodology uniquely integrates synchronized multimodal data—including multi-view RGB video from environmental cameras, close-up facial videos, and physiological signals (e.g., PPG, EDA) captured via smartwatches—across diverse ethnic groups. Annotation employs a dual-verification strategy: consensus labeling by four domain experts combined with student self-reports, supporting a fine-grained joint attentiveness–affect labeling scheme. Technically, the pipeline incorporates multi-view visual representation learning, temporal modeling of physiological time series, and high-precision cross-modal temporal alignment. This dataset constitutes the most comprehensive benchmark to date for classroom engagement analysis, bridging a critical gap in fine-grained behavioral understanding within real-world educational contexts and enabling joint, interpretable modeling of attention and emotion.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Security and Privacy: Large-scale security measurements

Abstract

In this paper, a novel dataset is introduced, designed to assess student attention within in-person classroom settings. This dataset encompasses RGB camera data, featuring multiple cameras per student to capture both posture and facial expressions, in addition to smartwatch sensor data for each individual. This dataset allows machine learning algorithms to be trained to predict attention and correlate it with emotion. A comprehensive suite of attention and emotion labels for each student is provided, generated through self-reporting as well as evaluations by four different experts. Our dataset uniquely combines facial and environmental camera data, smartwatch metrics, and includes underrepresented ethnicities in similar datasets, all within in-the-wild, in-person settings, making it the most comprehensive dataset of its kind currently available. The dataset presented offers an extensive and diverse collection of data pertaining to student interactions across different educational contexts, augmented with additional metadata from other tools. This initiative addresses existing deficiencies by offering a valuable resource for the analysis of student attention and emotion in face-to-face lessons.

Problem

• Recognize student engagement in classrooms
• Combine camera and smartwatch data for analysis
• Address dataset gaps with diverse ethnicity inclusion

Innovation

• Combines RGB and smartwatch data
• Includes diverse ethnic representation
• Utilizes multi-expert labeled dataset