Enhancing Facial Expression Recognition in Head-Mounted Displays with Synthetic Data

2026-07-05

AI Summary

This work addresses the challenges of facial expression recognition (FER) in head-mounted displays (HMDs), where unique camera viewpoints, scarcity of real-world data, and privacy concerns hinder performance. To overcome these limitations, the authors propose the first synthetic data generation framework tailored for HMD scenarios. The approach reconstructs textured 3D face meshes and employs a configurable virtual camera system to render images from head-mounted camera (HMC) perspectives. A Texture Space Alignment Network (TSAN) is introduced to preserve fine-grained expression details during rendering. This method effectively bridges the domain gap between synthetic and real data. Experiments on both simulated and real HMC datasets demonstrate that models trained solely on the generated synthetic data outperform those trained on existing real datasets and exhibit superior cross-device generalization across varying camera configurations.

Abstract

Facial expression recognition (FER) is crucial for social interaction in mixed reality environments that employ head-mounted displays (HMD). However, collecting FER data from head-mounted cameras (HMC) is challenging due to privacy concerns and the diversity of HMD platforms. Moreover, existing FER datasets are not directly applicable due to the unique perspectives of HMCs. The lack of sufficient data hinders the development of neural network-based HMC FER methods. To address data scarcity, we propose a data synthesis framework that generates HMC-view images from frontal-view images, leveraging abundant existing annotated datasets. Specifically, we first reconstruct 3D textured meshes from images and then apply a configurable camera system to render images from the HMC perspective. Additionally, we introduce a texture-space alignment network (TSAN) that enables accurate texture sampling from images to preserve detailed facial expressions. To evaluate the proposed method, we conduct extensive experiments on both simulated and real HMC datasets. Experimental results demonstrate that models trained on our synthetic dataset outperform those trained on existing datasets and exhibit better generalization across different camera configurations.

Problem

• Facial Expression Recognition
• Head-Mounted Displays
• Data Scarcity
• Synthetic Data
• Head-Mounted Cameras

Innovation

• synthetic data
• facial expression recognition
• head-mounted display
• texture-space alignment network
• 3D reconstruction