Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features

2026-07-23

AI Summary

This study addresses the limitations of conventional face recognition in unconstrained environments, where performance degrades due to expression variations, occlusions, and poor illumination, and dynamic identity cues remain underutilized. To overcome these challenges, the work proposes, for the first time, using valence-arousal (VA) estimation as a pretraining task to extract soft biometric features embedded in facial dynamics for identity recognition. A Quality-Aware Adaptive Fusion (QAAF) mechanism is introduced, integrating soft gating, quality-dependent dropout, and late fusion to dynamically adjust modality weights under conditions of missing or low-quality data. Experiments demonstrate that the method achieves a Concordance Correlation Coefficient (CCC) of 0.472 for VA estimation on Aff-wild2 and significantly enhances ArcFace performance on AFEW-VA and YouTube Faces, reducing false acceptance rates to 0.021 and 0.104, respectively, while correcting 68.2% of previously misaccepted samples.

Abstract

Conventional face recognition relies on static appearance cues and degrades in unconstrained settings with expression variation, occlusion, and poor lighting. We hypothesize that audiovisual expression dynamics carry identity-discriminative information complementary to static appearance, and that extracting this signal requires multimodal representations robust to the variable input quality of in-the-wild video. To learn such representations, we cast multimodal valence-arousal (VA) estimation as a pretext task and propose Quality-Aware Adaptive Fusion (QAAF), which estimates per-sample, per-modality reliability and adapts each modality's contribution through learned soft gating and a quality-dependent dropout. For the problem of VA estimation, QAAF achieves an average Concordance Correlation Coefficient (CCC) of 0.472 via late fusion ensembling on Aff-wild2, improving over a baseline ensemble under the same setting (0.415) as well as a single-backbone baseline (0.288). Furthermore, the proposed QAAF demonstrates greater resilience to unavailable modalities, with only a 7.5-34.4% relative decrease in CCC when one modality is missing. We then probe whether these VA-trained features encode identity without identity-specific training. On AFEW-VA (67 actors) and YTF (1,595 subjects), VA-trained backbone features rank first among evaluated soft biometric methods, and score-level fusion with ArcFace lowers EER on both datasets (0.022 to 0.021 on AFEW-VA, 0.106 to 0.104 on YTF), correcting 68.2% of ArcFace's false accepts on AFEW-VA. These findings establish multimodal VA estimation as a soft biometric modality complementary to conventional face recognition.

Problem

• face recognition
• multimodal fusion
• valence-aroussal
• soft biometrics
• in-the-wild video

Innovation

• Quality-Aware Adaptive Fusion
• Multimodal Fusion
• Valence-Arousal Estimation
• Soft Biometrics
• Pretext Task