🤖 AI Summary
This study addresses the challenge of recognizing ambivalence/hesitancy (A/H) states in video-recorded individuals by proposing a multimodal fusion approach based on inter-modal embedding discrepancies. The method explicitly models cross-modal inconsistency—a hallmark of A/H—by computing absolute differences among visual, audio, and textual embeddings in a shared latent space. Visual features are derived using Py-Feat to extract action units (AUs), audio representations are obtained via Wav2Vec 2.0, and textual inputs are encoded with BERT; these modalities are subsequently aligned through a BiLSTM combined with attention-based pooling. Evaluated on the BAH dataset, the model achieves a Macro F1 score of 0.6808, substantially outperforming baseline methods (0.2827). Further analysis confirms that temporal variability in AUs serves as a critical visual discriminative cue for A/H detection.
📝 Abstract
We address the Ambivalence/Hesitancy (A/H) Video Recognition Challenge at the 10th ABAW Competition (CVPR 2026). We propose a divergence-based multimodal fusion that explicitly measures cross-modal conflict between visual, audio, and textual channels. Visual features are encoded as Action Units (AUs) extracted via Py-Feat, audio via Wav2Vec 2.0, and text via BERT. Each modality is processed by a BiLSTM with attention pooling and projected into a shared embedding space. The fusion module computes pairwise absolute differences between modality embeddings, directly capturing the incongruence that characterizes A/H. On the BAH dataset, our approach achieves a Macro F1 of 0.6808 on the validation test set, outperforming the challenge baseline of 0.2827. Statistical analysis across 1{,}132 videos confirms that temporal variability of AUs is the dominant visual discriminator of A/H.