Roll Your Eyes: Gaze Redirection via Explicit 3D Eyeball Rotation

2025-08-08

AI Summary

Existing neural radiance field (NeRF)-based gaze redirection methods rely on implicit representations, making it difficult to explicitly model ocular rotation and translation—resulting in distorted and inaccurately directed gaze images. This paper proposes the first explicit 3D eyeball-structured gaze redirection framework. It employs 3D Gaussian splatting (3DGS) to accurately reconstruct eyeball geometry, introduces a differentiable explicit transformation module for rigid-eye motion modeling, and designs an adaptive deformation network to capture subtle extraocular muscle dynamics. For the first time, our approach integrates anatomically consistent, explicit eyeball geometry into gaze editing—balancing geometric fidelity and physiological plausibility. Evaluated on the ETH-XGaze dataset, our method surpasses state-of-the-art approaches in both image quality (PSNR/SSIM) and gaze angle estimation error. It enables high-fidelity, controllable novel-view gaze synthesis with precise anatomical grounding.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Embodied AI
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

We propose a novel 3D gaze redirection framework that leverages an explicit 3D eyeball structure. Existing gaze redirection methods are typically based on neural radiance fields, which employ implicit neural representations via volume rendering. Unlike these NeRF-based approaches, where the rotation and translation of 3D representations are not explicitly modeled, we introduce a dedicated 3D eyeball structure to represent the eyeballs with 3D Gaussian Splatting (3DGS). Our method generates photorealistic images that faithfully reproduce the desired gaze direction by explicitly rotating and translating the 3D eyeball structure. In addition, we propose an adaptive deformation module that enables the replication of subtle muscle movements around the eyes. Through experiments conducted on the ETH-XGaze dataset, we demonstrate that our framework is capable of generating diverse novel gaze images, achieving superior image quality and gaze estimation accuracy compared to previous state-of-the-art methods.

Problem

• Explicit 3D eyeball rotation for gaze redirection
• Improving realism with adaptive eye muscle movement
• Superior image quality and gaze accuracy vs NeRF

Innovation

• Explicit 3D eyeball structure for gaze redirection
• 3D Gaussian Splatting for photorealistic gaze images
• Adaptive deformation module for eye muscle movements