GlintMarkers: Spatial Perception on XR Eyewear using Corneal Reflections

2026-04-14

AI Summary

This work proposes a gaze-driven spatial awareness method leveraging the built-in cameras of XR glasses, which utilizes the cornea as a specular surface to capture faint reflections of environmental objects. By introducing near-infrared retroreflective markers to enhance signal strength and integrating a Perspective-n-Point (PnP) pose estimation algorithm specifically tailored to the imaging characteristics of the cornea, the system achieves, for the first time, the recovery of an object’s direction, distance, and unique identity from low-resolution corneal reflections. The approach demonstrates the feasibility of extracting spatial semantic information from corneal images under stringent resource constraints, establishing a novel paradigm for lightweight, sensor-free eye-tracking interaction in extended reality environments.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services

Abstract

We present GlintMarkers, the first system to perform gaze-driven spatial perception using the inward-facing cameras on XR eyewear. Our key observation is that the cornea acts as a mirror that encodes both gaze direction and visual information about the environment in a small, low-contrast reflection. To extract spatial and semantic information from this reflection despite the camera's limited pixel budget, we present a passive retroreflective marker design that concentrates reflected near-infrared light onto the cornea, producing bright glint patterns. We develop a custom Perspective-n-Point (PnP) estimation framework adapted to corneal imaging and perform orientation and distance estimation of tagged objects, as well as unique object identification.

Problem

• spatial perception
• corneal reflection
• XR eyewear
• gaze-driven
• object identification

Innovation

• corneal reflection
• XR eyewear
• gaze-driven spatial perception
• retroreflective markers
• Perspective-n-Point