Learning Spatio-Temporal Feature Representations for Video-Based Gaze Estimation

2025-12-19

AI Summary

Video-based gaze estimation requires joint modeling of spatial structure within the eye region and temporal dynamics across frames; however, existing methods typically decouple these aspects, leading to intra-frame detail loss and limited temporal modeling capacity. This paper proposes an end-to-end spatiotemporal joint representation learning framework. We innovatively treat eyelid features as a “spatial sequence” and employ recursive spatiotemporal modeling: first capturing intra-frame contextual relationships, then propagating dynamic changes across frames—thereby avoiding information degradation from early spatial pooling. The architecture integrates a CNN backbone with channel-wise attention and self-attention modules to enable hierarchical alignment between eye-region and facial features. Spatial serialization and temporal propagation structures are further introduced to enhance dynamic perception. Our method achieves state-of-the-art performance on the EVE dataset, supports zero-shot transfer and personalized adaptation, and significantly improves robustness and accuracy under unconstrained, commodity-camera settings.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Video-based gaze estimation methods aim to capture the inherently temporal dynamics of human eye gaze from multiple image frames. However, since models must capture both spatial and temporal relationships, performance is limited by the feature representations within a frame but also between multiple frames. We propose the Spatio-Temporal Gaze Network (ST-Gaze), a model that combines a CNN backbone with dedicated channel attention and self-attention modules to fuse eye and face features optimally. The fused features are then treated as a spatial sequence, allowing for the capture of an intra-frame context, which is then propagated through time to model inter-frame dynamics. We evaluated our method on the EVE dataset and show that ST-Gaze achieves state-of-the-art performance both with and without person-specific adaptation. Additionally, our ablation study provides further insights into the model performance, showing that preserving and modelling intra-frame spatial context with our spatio-temporal recurrence is fundamentally superior to premature spatial pooling. As such, our results pave the way towards more robust video-based gaze estimation using commonly available cameras.

Problem

• Learning spatio-temporal features for video gaze estimation
• Improving intra-frame and inter-frame feature representation
• Achieving robust gaze estimation with common cameras

Innovation

• CNN backbone with attention modules for feature fusion
• Spatial sequence modeling for intra-frame context capture
• Temporal propagation to model inter-frame dynamics