Enhancing Screen Time Identification in Children with a Multi-View Vision Language Model and Screen Time Tracker

2024-10-02

arXiv.org

AI Summary

Current screen exposure monitoring for young children relies on subjective self-reports or bulky wearable devices, hindering accurate, unobtrusive, long-term quantification. This paper introduces the first lightweight wearable system for child-centric screen time tracking, integrating egocentric video with a multi-view vision-language model (VLM). Our approach employs temporal egocentric modeling, multimodal sensor fusion, and self-supervised temporal representation learning to dynamically detect real-world screen use behaviors. Key innovations include: (i) the first multi-view VLM architecture explicitly modeling contextual cues of screen interaction; and (ii) a hardware-algorithm co-designed Screen Time Tracker framework. Evaluated on a naturalistic child free-play dataset, our method achieves 23.6% higher screen time detection accuracy than single-view VLM and object-detection baselines. It enables fine-grained, long-term, passive monitoring in ecologically valid settings.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Being able to accurately monitor the screen exposure of young children is important for research on phenomena linked to screen use such as childhood obesity, physical activity, and social interaction. Most existing studies rely upon self-report or manual measures from bulky wearable sensors, thus lacking efficiency and accuracy in capturing quantitative screen exposure data. In this work, we developed a novel sensor informatics framework that utilizes egocentric images from a wearable sensor, termed the screen time tracker (STT), and a vision language model (VLM). In particular, we devised a multi-view VLM that takes multiple views from egocentric image sequences and interprets screen exposure dynamically. We validated our approach by using a dataset of children's free-living activities, demonstrating significant improvement over existing methods in plain vision language models and object detection models. Results supported the promise of this monitoring approach, which could optimize behavioral research on screen exposure in children's naturalistic settings.

Problem

• Accurately monitor children's screen exposure for health research
• Overcome inefficiency of self-report and bulky wearable sensors
• Improve dynamic screen exposure detection using multi-view VLM

Innovation

• Utilizes egocentric images from wearable screen time tracker
• Employs multi-view vision language model for dynamic interpretation
• Validated with children's free-living activities dataset