GazeGaussian: High-Fidelity Gaze Redirection with 3D Gaussian Splatting

2024-11-20

arXiv.org

AI Summary

Out-of-distribution (OOD) gaze estimation suffers from poor generalization, loss of fine facial details, and high computational overhead. To address these challenges, we propose a dual-stream 3D Gaussian Splatting (3DGS) framework that decouples holistic face modeling from localized eye modeling—marking the first 3D Gaussian representation explicitly designed for gaze-controllable, disentangled eye regions. We introduce an expression-conditioned guidance module to enhance cross-subject generalization and integrate rigid eyeball geometry modeling to enable precise, target-direction-driven ocular rotation control. Evaluated across multiple datasets, our method achieves over 10× faster rendering than NeRF-based approaches, reduces gaze redirection error by 32%, significantly improves facial detail fidelity, and substantially boosts the OOD generalization performance of downstream gaze estimators.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Gaze estimation encounters generalization challenges when dealing with out-of-distribution data. To address this problem, recent methods use neural radiance fields (NeRF) to generate augmented data. However, existing methods based on NeRF are computationally expensive and lack facial details. 3D Gaussian Splatting (3DGS) has become the prevailing representation of neural fields. While 3DGS has been extensively examined in head avatars, it faces challenges with accurate gaze control and generalization across different subjects. In this work, we propose GazeGaussian, a high-fidelity gaze redirection method that uses a two-stream 3DGS model to represent the face and eye regions separately. By leveraging the unstructured nature of 3DGS, we develop a novel eye representation for rigid eye rotation based on the target gaze direction. To enhance synthesis generalization across various subjects, we integrate an expression-conditional module to guide the neural renderer. Comprehensive experiments show that GazeGaussian outperforms existing methods in rendering speed, gaze redirection accuracy, and facial synthesis across multiple datasets. We also demonstrate that existing gaze estimation methods can leverage GazeGaussian to improve their generalization performance. The code will be available at: https://ucwxb.github.io/GazeGaussian/.

Problem

• Overcome gaze estimation generalization issues with out-of-distribution data
• Address computational cost and detail loss in NeRF-based gaze redirection
• Improve cross-subject gaze control accuracy using 3D Gaussian Splatting

Innovation

• Two-stream 3DGS model for face and eyes
• Novel eye representation for gaze direction
• Expression-guided module for subject generalization