Causal Supervision of Attention for Affective Behaviour Analysis

2026-07-13

AI Summary

This study addresses the challenge that facial expression recognition in real-world scenarios is hindered by confounding factors such as identity, illumination, and head pose, which impair the generalization of affective models. To mitigate this, the authors propose a causally supervised attention mechanism that leverages cross-covariance independence regularization between Key and Value features, combined with SwiGLU nonlinear transformations, to steer the model toward attending to emotion-relevant regions invariant across individuals. This approach enables learning subject-invariant yet highly expressive representations. Evaluated on the s-Aff-Wild2 validation set, the method achieves a composite P-score of 1.2214 across three tasks—valence-arousal estimation (CCC = 0.5123), expression recognition (F1 = 0.3116), and action unit detection (F1 = 0.3974)—demonstrating significant improvements in multitask affective analysis performance.

Abstract

Affective Behaviour Analysis aims to enable machines to infer human affective states from behavioural signals, particularly facial expressions, in real-world environments. The \textit{11th Affective Behaviour Analysis in-the-wild Competition} includes the Multi-Task Learning Challenge based on the s-Aff-Wild2 database, where participants develop a unified framework for Valence-Arousal Estimation, Expression Recognition, and Action Unit Detection. This is challenging because emotion-related cues must be distinguished from spurious factors such as identity, illumination, pose, and demographic variation. Attention mechanisms are well suited as they aggregate information from the most informative facial regions, but may still exploit dataset-specific correlations instead of true affective cues. To improve generalization, we propose an attention pooling framework that promotes subject-invariant attention while increasing feature expressiveness. Our method consists of three components. First, we introduce causal supervision to enforce attention on facial regions with invariant predictive value across subjects. Second, we apply a cross-covariance independence regularization between Key (K) and Value (V) projections to encourage complementary, non-redundant representations. Finally, we replace the linear Value projection with a gated nonlinear SwiGLU transformation to increase feature expressiveness and capture finer-grained affective cues. Our method achieves $CCC_{VA}=0.5123$ for VA estimation on the official validation set, together with $F1_{EX}=0.3116$ and $F1_{AU}=0.3974$ for expression recognition and action unit detection, respectively, resulting in an overall $P$ score (the sum of the individual task metrics) of $1.2214$.

Problem

• Affective Behaviour Analysis
• Attention Mechanism
• Spurious Correlations
• In-the-wild Emotion Recognition
• Subject-invariant Representation

Innovation

• causal supervision
• attention mechanism
• invariant representation
• cross-covariance regularization
• SwiGLU