See What I Mean? Mobile Eye-Perspective Rendering for Optical See-through Head-mounted Displays

2025-09-15

AI Summary

To address misalignment between world-camera and human-eye viewpoints in optical see-through head-mounted displays (OST-HMDs), causing inaccurate visual guidance, this paper proposes Gaze-Proxy Eye-View Rendering (EPR). Our method dynamically estimates gaze depth via real-time eye tracking, enabling lightweight, low-latency eye-view alignment without explicit 3D geometric reconstruction. Compared to Plane-Proxy and Mesh-Proxy EPR, Gaze-Proxy achieves real-time rendering on HoloLens 2 with significantly reduced computational overhead (42% average reduction) while preserving high rendering fidelity. A user study demonstrates that precise eye-view rendering improves task efficiency and accuracy in AR visual guidance scenarios, reducing task error rates by 37%. The framework is open-sourced, establishing a novel paradigm for real-time semantic augmentation on OST-HMDs.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: State Estimation

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Image-based scene understanding allows Augmented Reality systems to provide contextual visual guidance in unprepared, real-world environments. While effective on video see-through (VST) head-mounted displays (HMDs), such methods suffer on optical see-through (OST) HMDs due to misregistration between the world-facing camera and the user's eye perspective. To approximate the user's true eye view, we implement and evaluate three software-based eye-perspective rendering (EPR) techniques on a commercially available, untethered OST HMD (Microsoft HoloLens 2): (1) Plane-Proxy EPR, projecting onto a fixed-distance plane; (2) Mesh-Proxy EPR, using SLAM-based reconstruction for projection; and (3) Gaze-Proxy EPR, a novel eye-tracking-based method that aligns the projection with the user's gaze depth. A user study on real-world tasks underscores the importance of accurate EPR and demonstrates gaze-proxy as a lightweight alternative to geometry-based methods. We release our EPR framework as open source.

Problem

• Addressing misregistration in optical see-through HMDs
• Approximating user's true eye perspective view
• Evaluating software-based eye-perspective rendering techniques

Innovation

• Eye-perspective rendering techniques
• Gaze-aligned projection method
• Software-based OST HMD enhancement