Action Unit Enhance Dynamic Facial Expression Recognition

2025-07-10

AI Summary

To address two key challenges in dynamic facial expression recognition (DFER)—insufficient exploitation of action unit (AU) prior knowledge and severe label imbalance—this paper proposes AU-DFER, a novel framework. Methodologically, it is the first to systematically model the quantified AU-expression mapping as a learnable AU weight matrix and integrates it into temporal feature learning via an AU-aware loss function—without increasing inference overhead. Furthermore, it introduces an AU-loss-based label reweighting strategy to mitigate class bias toward minority expressions (e.g., “disgust”, “fear”) on benchmark datasets such as DFEW and FERV39K. Extensive experiments across multiple DFER benchmarks and state-of-the-art backbone models demonstrate consistent performance gains: AU-DFER achieves significant improvements in overall accuracy, with particularly notable enhancements of 5.2–8.7% for low-frequency expressions, surpassing existing SOTA methods.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Dynamic Facial Expression Recognition(DFER) is a rapidly evolving field of research that focuses on the recognition of time-series facial expressions. While previous research on DFER has concentrated on feature learning from a deep learning perspective, we put forward an AU-enhanced Dynamic Facial Expression Recognition architecture, namely AU-DFER, that incorporates AU-expression knowledge to enhance the effectiveness of deep learning modeling. In particular, the contribution of the Action Units(AUs) to different expressions is quantified, and a weight matrix is designed to incorporate a priori knowledge. Subsequently, the knowledge is integrated with the learning outcomes of a conventional deep learning network through the introduction of AU loss. The design is incorporated into the existing optimal model for dynamic expression recognition for the purpose of validation. Experiments are conducted on three recent mainstream open-source approaches to DFER on the principal datasets in this field. The results demonstrate that the proposed architecture outperforms the state-of-the-art(SOTA) methods without the need for additional arithmetic and generally produces improved results. Furthermore, we investigate the potential of AU loss function redesign to address data label imbalance issues in established dynamic expression datasets. To the best of our knowledge, this is the first attempt to integrate quantified AU-expression knowledge into various DFER models. We also devise strategies to tackle label imbalance, or minor class problems. Our findings suggest that employing a diverse strategy of loss function design can enhance the effectiveness of DFER. This underscores the criticality of addressing data imbalance challenges in mainstream datasets within this domain. The source code is available at https://github.com/Cross-Innovation-Lab/AU-DFER.

Problem

• Enhancing dynamic facial expression recognition using AU knowledge
• Quantifying AU contributions to expressions via weight matrix
• Addressing data label imbalance with AU loss redesign

Innovation

• AU-enhanced deep learning for expression recognition
• Weight matrix integrates AU-expression knowledge
• AU loss redesign tackles label imbalance