Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

2026-06-29

AI Summary

Existing video Transformers struggle to model subtle local dynamics in facial expression understanding due to their self-attention mechanisms’ inherent bias toward global motion and coarse-grained temporal variations. To address this limitation, this work proposes MiRA, a novel framework that introduces frame-level confidence and intra-frame concentration statistics derived directly from self-attention maps, enabling attention redistribution without any additional trainable parameters. MiRA integrates precise post-softmax reallocation with a lightweight pre-softmax approximation and further incorporates an efficient flashLite mode into the FlashAttention kernel. Evaluated on multiple challenging facial expression recognition benchmarks, MiRA significantly outperforms strong Vision Transformer baselines, demonstrating enhanced spatiotemporal selectivity in capturing fine-grained facial dynamics.

Abstract

Understanding facial expressions in videos requires modeling subtle and localized facial dynamics under unconstrained conditions. Although recent Vision Transformer~(ViT)-based video models have shown strong performance through large-scale self-supervised pretraining, their attention mechanisms often emphasize dominant global motions and coarse temporal dynamics, limiting sensitivity to fine-grained facial variations. To address this limitation, we propose MiRA (Marginal-induced Attention Redistribution), a plug-in frame-marginal attention redistribution framework for ViT backbones that enhances spatio-temporal selectivity toward subtle facial dynamics without introducing additional trainable parameters. MiRA derives frame-level confidence and intra-frame concentration statistics from self-attention maps to estimate frame-wise marginal importance and redistribute attention toward spatiotemporally localized facial cues. We first introduce a principled \textit{exact mode} based on post-softmax attention redistribution. To further improve efficiency, we propose \textit{flashLite mode}, a lightweight pre-softmax approximation that integrates frame-marginal redistribution into FlashAttention kernels while preserving the effectiveness of the exact formulation. Experimental results on challenging Facial Expression Recognition~(FER) benchmarks demonstrate consistent improvements over strong ViT baselines.

Problem

• Facial Expression Recognition
• Video Transformers
• Attention Mechanism
• Fine-grained Facial Dynamics
• Spatio-temporal Modeling

Innovation

• MiRA
• Frame-marginal Attention Redistribution
• Video Transformers
• Facial Expression Recognition
• FlashAttention