Low Latency Gaze Tracking via Latent Optical Sensing

2026-05-18

AI Summary

This work addresses the limitations of conventional camera-based gaze tracking systems, which suffer from high latency, bandwidth demands, and computational overhead that hinder real-time human-computer interaction. The authors propose a task-driven, fully passive optical encoding scheme that leverages a microlens array and a co-designed binary chromium mask to achieve spatially multiplexed optical encoding. This approach directly extracts gaze-relevant latent features in the optical domain, eliminating the need for image capture and transmission. Using only a 4×4 phototransistor array paired with a lightweight neural network, the system achieves competitive gaze estimation accuracy on both simulated and real-world data, with an end-to-end latency as low as 3.4 milliseconds—significantly outperforming existing methods while substantially improving energy efficiency.

Abstract

We present a real-time gaze tracking system that directly acquires task-relevant latent features using a fully passive optical encoder. Instead of forming and processing full-resolution images, our approach leverages a microlens array with a co-designed binary chromium mask to perform spatially multiplexed optical encoding, producing a compact set of measurements sufficient for gaze estimation. By integrating sensing and feature extraction in the optical domain, the proposed system eliminates the need for high-bandwidth image readout and substantially reduces computational overhead. The encoded measurements are captured by a 4 x 4 phototransistor array and mapped to gaze direction using a lightweight neural network. Our proof-of-concept prototype enables an end-to-end sensing-to-inference latency of 3.4 ms, outperforming published research systems. We demonstrate the effectiveness of our approach on both simulated and real-world data, achieving competitive gaze estimation accuracy while significantly improving latency and energy efficiency compared to conventional camera-based pipelines. This work highlights the potential of task-driven optical sensing for ultra-low-latency, computationally efficient human-computer interaction systems.

Problem

• gaze tracking
• low latency
• optical sensing
• energy efficiency
• human-computer interaction

Innovation

• optical encoding
• gaze tracking
• latent sensing
• low latency
• task-driven optics