egoPPG: Heart Rate Estimation from Eye-Tracking Cameras in Egocentric Systems to Benefit Downstream Vision Tasks

2025-02-28

AI Summary

This work introduces egocentric photoplethysmography (PPG), a novel task for estimating the wearer’s heart rate (HR) non-invasively from built-in eye-tracking videos in first-person vision systems—requiring no additional hardware. To address this, we propose EgoPulseFormer, an end-to-end Transformer architecture that jointly performs dynamic periorbital region-of-interest (ROI) extraction and PPG waveform inversion, with multi-modal ground-truth alignment using synchronized electrocardiogram (ECG) and contact-based blood volume pulse (BVP) signals. Evaluated on a 13+ hour real-world dataset, our method achieves a mean absolute error of 8.82 bpm (r = 0.81) in HR estimation. Furthermore, incorporating the estimated HR signal improves downstream action proficiency recognition accuracy by 14%. This work establishes the first paradigm for physiological sensing directly from egocentric eye-movement video, significantly enhancing contextual understanding in first-person vision systems.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Large-scale security measurements

Abstract

Egocentric vision systems aim to understand the spatial surroundings and the wearer's behavior inside it, including motions, activities, and interaction with objects. Since a person's attention and situational responses are influenced by their physiological state, egocentric systems must also detect this state for better context awareness. In this paper, we propose egoPPG, a novel task for egocentric vision systems to extract a person's heart rate (HR) as a key indicator of the wearer's physiological state from the system's built-in sensors (e.g., eye tracking videos). We then propose EgoPulseFormer, a method that solely takes eye-tracking video as input to estimate a person's photoplethysmogram (PPG) from areas around the eyes to track HR values-without requiring additional or dedicated hardware. We demonstrate the downstream benefit of EgoPulseFormer on EgoExo4D, where we find that augmenting existing models with tracked HR values improves proficiency estimation by 14%. To train and validate EgoPulseFormer, we collected a dataset of 13+ hours of eye-tracking videos from Project Aria and contact-based blood volume pulse signals as well as an electrocardiogram (ECG) for ground-truth HR values. 25 participants performed diverse everyday activities such as office work, cooking, dancing, and exercising, which induced significant natural motion and HR variation (44-164 bpm). Our model robustly estimates HR (MAE=8.82 bpm) and captures patterns (r=0.81). Our results show how egocentric systems may unify environmental and physiological tracking to better understand user actions and internal states.

Problem

• Estimating heart rate from eye-tracking cameras
• Enhancing egocentric vision systems with physiological data
• Improving downstream tasks using heart rate information

Innovation

• Estimates heart rate from eye-tracking videos.
• Uses EgoPulseFormer for photoplethysmogram extraction.
• Enhances model proficiency with HR data.