3D Prior is All You Need: Cross-Task Few-shot 2D Gaze Estimation

2025-02-06

AI Summary

This work addresses the challenging problem of cross-device few-shot 2D gaze estimation, where only a handful (e.g., several) calibration images are available per device, and the screen pose is unknown. Key difficulties include domain shift from pre-trained 3D gaze models to 2D estimation, uncalibrated screen geometry, and extreme data scarcity. To tackle these, we propose a lightweight adaptation framework that requires no modification to the frozen pre-trained 3D gaze model. Our method: (1) introduces a physically interpretable, differentiable projection module to jointly estimate the screen’s coordinate transformation matrix; (2) employs a dynamic pseudo-labeling strategy with geometrically consistent flip augmentations applied in 3D space—avoiding ambiguity inherent in 2D label transformations; and (3) generates pseudo-labels via inverse 3D→2D projection for efficient few-shot fine-tuning. Extensive evaluation on MPIIGaze (laptop), EVE (desktop), and GazeCapture (mobile) demonstrates substantial improvements over state-of-the-art few-shot methods, confirming strong generalization across devices and practical deployability.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

3D and 2D gaze estimation share the fundamental objective of capturing eye movements but are traditionally treated as two distinct research domains. In this paper, we introduce a novel cross-task few-shot 2D gaze estimation approach, aiming to adapt a pre-trained 3D gaze estimation network for 2D gaze prediction on unseen devices using only a few training images. This task is highly challenging due to the domain gap between 3D and 2D gaze, unknown screen poses, and limited training data. To address these challenges, we propose a novel framework that bridges the gap between 3D and 2D gaze. Our framework contains a physics-based differentiable projection module with learnable parameters to model screen poses and project 3D gaze into 2D gaze. The framework is fully differentiable and can integrate into existing 3D gaze networks without modifying their original architecture. Additionally, we introduce a dynamic pseudo-labelling strategy for flipped images, which is particularly challenging for 2D labels due to unknown screen poses. To overcome this, we reverse the projection process by converting 2D labels to 3D space, where flipping is performed. Notably, this 3D space is not aligned with the camera coordinate system, so we learn a dynamic transformation matrix to compensate for this misalignment. We evaluate our method on MPIIGaze, EVE, and GazeCapture datasets, collected respectively on laptops, desktop computers, and mobile devices. The superior performance highlights the effectiveness of our approach, and demonstrates its strong potential for real-world applications.

Problem

• Adapt 3D gaze network for 2D prediction.
• Bridge gap between 3D and 2D gaze.
• Overcome challenges with limited training data.

Innovation

• Physics-based differentiable projection module
• Dynamic pseudo-labelling strategy
• Dynamic transformation matrix compensation