Towards Emotion Recognition with 3D Pointclouds Obtained from Facial Expression Images

2026-03-29

AI Summary

This study addresses the privacy risks and limitations of continuous real-time monitoring inherent in traditional 2D facial expression recognition. To overcome these challenges, the authors propose a privacy-preserving emotion recognition framework based on high-frequency wireless sensing (HFWS), which captures 3D facial point clouds without exposing visual imagery. They introduce AffectNet3D, the first 3D point cloud dataset derived from existing 2D benchmarks like AffectNet, by leveraging the FLAME facial model for conversion. Integrating point cloud cropping with a PointNet++ architecture, the system enables efficient emotion classification under wearable-device simulation conditions. Experimental results demonstrate that fine-tuning on BU-3DFE achieves over 70% accuracy, and remarkably, using only 25% of the annotated data surpasses the performance of full-dataset training, confirming the method’s efficacy and practical feasibility.

Abstract

Facial Emotion Recognition is a critical research area within Affective Computing due to its wide-ranging applications in Human Computer Interaction, mental health assessment and fatigue monitoring. Current FER methods predominantly rely on Deep Learning techniques trained on 2D image data, which pose significant privacy concerns and are unsuitable for continuous, real-time monitoring. As an alternative, we propose High-Frequency Wireless Sensing (HFWS) as an enabler of continuous, privacy-aware FER, through the generation of detailed 3D facial pointclouds via on-person sensors embedded in wearables. We present arguments supporting the privacy advantages of HFWS over traditional 2D imaging, particularly under increasingly stringent data protection regulations. A major barrier to adopting HFWS for FER is the scarcity of labeled 3D FER datasets. Towards addressing this issue, we introduce a FLAME-based method to generate 3D facial pointclouds from existing public 2D datasets. Using this approach, we create AffectNet3D, a 3D version of the AffectNet database. To evaluate the quality and usability of the generated data, we design a pointcloud refinement pipeline focused on isolating the facial region, and train the popular PointNet++ model on the refined pointclouds. Fine-tuning the model on a small subset of the unseen 3D FER dataset BU-3DFE yields a classification accuracy exceeding 70%, comparable to oracle-level performance. To further investigate the potential of HFWS-based FER for continuous monitoring, we simulate wearable sensing conditions by masking portions of the generated pointclouds. Experimental results show that models trained on AffectNet3D and fine-tuned with just 25% of BU-3DFE outperform those trained solely on BU-3DFE. These findings highlight the viability of our pipeline and support the feasibility of continuous, privacy-aware FER via wearable HFWS systems.

Problem

• Facial Emotion Recognition
• Privacy
• 3D Pointclouds
• Wearable Sensing
• Dataset Scarcity

Innovation

• 3D pointcloud
• Facial Emotion Recognition
• High-Frequency Wireless Sensing
• FLAME model
• privacy-aware sensing