Gaze Prediction in Virtual Reality Without Eye Tracking Using Visual and Head Motion Cues

2026-01-26

AI Summary

This work proposes a lightweight multimodal approach to accurately predict user gaze direction in virtual reality scenarios where eye-tracking hardware is unavailable or restricted by privacy constraints—a critical capability for techniques such as foveated rendering. The method uniquely integrates head-mounted display (HMD) motion signals with visual saliency cues from video frames by leveraging UniSal for visual feature extraction and combining TSMixer with LSTM to construct a temporal prediction module. Experiments on the EHTask dataset and commercial VR devices demonstrate that the proposed approach significantly outperforms baseline methods such as Center-of-HMD and Mean Gaze, achieving high prediction accuracy, low latency, and practical deployability without requiring eye-tracking data, thereby enhancing the naturalness and efficiency of VR interactions.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Gaze prediction plays a critical role in Virtual Reality (VR) applications by reducing sensor-induced latency and enabling computationally demanding techniques such as foveated rendering, which rely on anticipating user attention. However, direct eye tracking is often unavailable due to hardware limitations or privacy concerns. To address this, we present a novel gaze prediction framework that combines Head-Mounted Display (HMD) motion signals with visual saliency cues derived from video frames. Our method employs UniSal, a lightweight saliency encoder, to extract visual features, which are then fused with HMD motion data and processed through a time-series prediction module. We evaluate two lightweight architectures, TSMixer and LSTM, for forecasting future gaze directions. Experiments on the EHTask dataset, along with deployment on commercial VR hardware, show that our approach consistently outperforms baselines such as Center-of-HMD and Mean Gaze. These results demonstrate the effectiveness of predictive gaze modeling in reducing perceptual lag and enhancing natural interaction in VR environments where direct eye tracking is constrained.

Problem

• gaze prediction
• virtual reality
• eye tracking
• visual saliency
• head motion

Innovation

• gaze prediction
• visual saliency
• head motion cues
• foveated rendering
• VR interaction