A Sleep Monitoring System Based on Audio, Video and Depth Information

2025-12-05

AI Summary

Addressing the challenge of non-invasive, quantitative assessment of sleep disturbances in home environments, this paper proposes an event-driven multimodal monitoring framework integrating audio, video, and depth data. Synchronized data are acquired using an infrared depth sensor, an RGB-D camera, and a quad-microphone array. A dual-stream background modeling approach—combining depth and RGB modalities—is developed to separately characterize human motion and ambient illumination changes. Temporal analysis of depth sequences and acoustic event detection further enable precise identification of three disturbance types: body movement, light switching, and environmental noise. Evaluated under low-illumination, real-world sleep conditions, the framework demonstrates significantly improved detection accuracy and robust scene adaptability. It provides a practical, contactless solution for home-based sleep quality assessment.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Large-scale security measurements

Abstract

For quantitative evaluation of sleep disturbances, a noninvasive monitoring system is developed by introducing an event-based method. We observe sleeping in home context and classify the sleep disturbances into three types of events: motion events, light-on/off events and noise events. A device with an infrared depth sensor, a RGB camera, and a four-microphone array is used in sleep monitoring in an environment with barely light sources. One background model is established in depth signals for measuring magnitude of movements. Because depth signals cannot observe lighting changes, another background model is established in color images for measuring magnitude of lighting effects. An event detection algorithm is used to detect occurrences of events from the processed data of the three types of sensors. The system was tested in sleep condition and the experiment result validates the system reliability.

Problem

• Develops a noninvasive system for monitoring sleep disturbances
• Classifies disturbances into motion, light, and noise events
• Uses audio, video, and depth sensors in low-light conditions

Innovation

• Uses audio, video, and depth sensors for sleep monitoring
• Applies event-based method to classify sleep disturbances
• Establishes separate background models for movement and lighting