🤖 AI Summary
This work addresses the challenges of frame-level eight-class facial expression recognition in unconstrained videos, including inaccurate localization, pose and scale variations, motion blur, and temporal inconsistency. To tackle these issues, a two-stage audio-visual multimodal approach is proposed. In the first stage, robust visual features are extracted using DINOv2, augmented with padding-aware data augmentation and processed by a mixture-of-experts classification head. The second stage integrates multi-scale re-cropped visual features with Wav2Vec 2.0 audio representations through a lightweight gated fusion module, further enhanced by temporal smoothing during inference to improve stability. The method achieves a Macro-F1 score of 0.5368 on the ABAW validation set and 0.5122 ± 0.0277 under five-fold cross-validation, significantly outperforming the official baseline.
📝 Abstract
This paper addresses the expression (EXPR) recognition challenge in the 10th Affective Behavior Analysis in-the-Wild (ABAW) workshop and competition, which requires frame-level classification of eight facial emotional expressions from unconstrained videos. This task is challenging due to inaccurate face localization, large pose and scale variations, motion blur, temporal instability, and other confounding factors across adjacent frames. We propose a two-stage dual-modal (audio-visual) model to address these difficulties. Stage I focuses on robust visual feature extraction with a pretrained DINOv2-based encoder. Specifically, DINOv2 ViT-L/14 is used as the backbone, a padding-aware augmentation (PadAug) strategy is employed for image padding and data preprocessing from raw videos, and a mixture-of-experts (MoE) training head is introduced to enhance classifier diversity. Stage II addresses modality fusion and temporal consistency. For the visual modality, faces are re-cropped from raw videos at multiple scales, and the extracted visual features are averaged to form a robust frame-level representation. Concurrently, frame-aligned Wav2Vec 2.0 audio features are derived from short audio windows to provide complementary acoustic cues. These dual-modal features are integrated via a lightweight gated fusion module, followed by inference-time temporal smoothing. Experiments on the ABAW dataset demonstrate the effectiveness of the proposed method. The two-stage model achieves a Macro-F1 score of 0.5368 on the official validation set and 0.5122 +/- 0.0277 under 5-fold cross-validation, outperforming the official baselines.