OFERA: Blendshape-driven 3D Gaussian Control for Occluded Facial Expression to Realistic Avatars in VR

2026-02-02

AI Summary

Current VR headsets rely on additional sensors or built-in cameras to drive facial expressions, suffering from issues such as bulky hardware, privacy concerns, and limited data availability. This work proposes the first end-to-end blendshape-driven framework that leverages only the blendshape signals output by commercial VR headsets to achieve real-time, photorealistic avatar animation under occlusion. By integrating an expressive mapping module with 3D Gaussian avatar training, the method introduces blendshape distribution alignment via linear regression, a parameter mapper, and joint optimization with Gaussian splatting rendering—ensuring consistent expression distributions without requiring extra hardware. Experiments demonstrate superior quantitative performance over existing mapping approaches, and user studies confirm significant improvements in expression fidelity, avatar realism, and remote presence experience.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Security and Privacy: Data transparency and provenance

Abstract

We propose OFERA, a novel framework for real-time expression control of photorealistic Gaussian head avatars for VR headset users. Existing approaches attempt to recover occluded facial expressions using additional sensors or internal cameras, but sensor-based methods increase device weight and discomfort, while camera-based methods raise privacy concerns and suffer from limited access to raw data. To overcome these limitations, we leverage the blendshape signals provided by commercial VR headsets as expression inputs. Our framework consists of three key components: (1) Blendshape Distribution Alignment (BDA), which applies linear regression to align the headset-provided blendshape distribution to a canonical input space; (2) an Expression Parameter Mapper (EPM) that maps the aligned blendshape signals into an expression parameter space for controlling Gaussian head avatars; and (3) a Mapper-integrated Avatar (MiA) that incorporates EPM into the avatar learning process to ensure distributional consistency. Furthermore, OFERA establishes an end-to-end pipeline that senses and maps expressions, updates Gaussian avatars, and renders them in real-time within VR environments. We show that EPM outperforms existing mapping methods on quantitative metrics, and we demonstrate through a user study that the full OFERA framework enhances expression fidelity while preserving avatar realism. By enabling real-time and photorealistic avatar expression control, OFERA significantly improves telepresence in VR communication. A project page is available at https://ysshwan147.github.io/projects/ofera/.

Problem

• occluded facial expression
• realistic avatars
• VR
• blendshape-driven
• telepresence

Innovation

• Blendshape-driven
• 3D Gaussian Avatars
• Occluded Facial Expression
• Real-time VR Telepresence
• Expression Parameter Mapping