Autonomous AI Surveillance: Multimodal Deep Learning for Cognitive and Behavioral Monitoring

2025-07-02

AI Summary

This study addresses the challenge of automated classroom student attention and behavioral assessment. We propose a multimodal end-to-end intelligent monitoring framework that integrates face identity recognition (LResNet + MTCNN), mobile phone usage detection, and drowsiness behavior analysis (both based on YOLOv8), deployed on an ESP32-CAM edge acquisition platform and a PHP-based web system for real-time processing and feedback. Our key contribution lies in synergistic multi-model modeling of student attentiveness, enabling simultaneous automated attendance tracking and fine-grained behavioral cognition analysis. Experimental results demonstrate strong performance: 97.42% mAP@50 for drowsiness detection, 86.45% accuracy for face recognition, and 85.89% mAP@50 for mobile phone usage detection. The system exhibits significant advantages in accuracy, real-time responsiveness, and scalability, offering a practical, deployable solution for intelligent classroom management.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This study presents a novel classroom surveillance system that integrates multiple modalities, including drowsiness, tracking of mobile phone usage, and face recognition,to assess student attentiveness with enhanced precision.The system leverages the YOLOv8 model to detect both mobile phone and sleep usage,(Ghatge et al., 2024) while facial recognition is achieved through LResNet Occ FC body tracking using YOLO and MTCNN.(Durai et al., 2024) These models work in synergy to provide comprehensive, real-time monitoring, offering insights into student engagement and behavior.(S et al., 2023) The framework is trained on specialized datasets, such as the RMFD dataset for face recognition and a Roboflow dataset for mobile phone detection. The extensive evaluation of the system shows promising results. Sleep detection achieves 97. 42% mAP@50, face recognition achieves 86. 45% validation accuracy and mobile phone detection reach 85. 89% mAP@50. The system is implemented within a core PHP web application and utilizes ESP32-CAM hardware for seamless data capture.(Neto et al., 2024) This integrated approach not only enhances classroom monitoring, but also ensures automatic attendance recording via face recognition as students remain seated in the classroom, offering scalability for diverse educational environments.(Banada,2025)

Problem

• Develops AI system to monitor student attentiveness in classrooms
• Integrates drowsiness, phone usage, and face recognition for precision
• Uses YOLOv8 and LResNet for real-time behavioral tracking

Innovation

• Multimodal deep learning for student monitoring
• YOLOv8 and LResNet for object and face recognition
• PHP web app with ESP32-CAM for real-time data