Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition

2026-07-14

AI Summary

This work addresses the challenge of recognizing hesitant or conflicted psychological states of speakers in videos using only audio and textual modalities, without relying on visual cues. The proposed approach segments input into overlapping 5-second windows and integrates prosodic audio features (320-dimensional), sentiment-aware RoBERTa text embeddings, and handcrafted psycholinguistic support features (74-dimensional). Cross-modal fusion is achieved through temporal cross-attention, and the psycholinguistic support features are incorporated prior to gated multiple instance learning pooling to dynamically modulate window-level weights. The key innovation lies in the novel integration of psycholinguistic support features with an audio-text cross-attention mechanism for vision-free emotion recognition. Evaluated on the ABAW challenge development set, an ensemble of five models achieves an average accuracy of 0.875 and a macro F1-score of 0.72.

Abstract

We present an audio-text system for the Ambivalence/Hesitancy Video Recognition Challenge of the 11th ABAW Competition. The method excludes visual frames and represents each video as overlapping 5-second windows aligned with transcript timestamps. Each window combines a 320-dimensional prosodic audio descriptor, a 768-dimensional emotion-oriented RoBERTa embedding, and 74 handcrafted features capturing uncertainty, hedging, and attitudinal conflict. Audio and text are fused via temporal cross-attention, while support features are injected prior to gated multiple-instance learning (MIL) pooling to modulate the window's importance. Predictions from five independently initialized models are averaged. On the labeled public development set, the ensemble achieved an average precision of 0.875 and a macro-F1 of 0.72. Our source code is publicly available at https://github.com/Liga-de-IA-PUCPR/abaw-11-ah-challenge/.

Problem

• ambivalence
• hesitancy
• audio-text recognition
• psycholinguistic features
• uncertainty detection

Innovation

• audio-text cross-attention
• psycholinguistic features
• gated MIL pooling
• ambivalence recognition
• prosodic-emotion fusion