A Calibrated Multimodal Ensemble for Ambivalence/Hesitancy Recognition: System Description and Private-Test Submission Strategy

2026-07-13

AI Summary

This work addresses the challenge of automatically detecting users’ ambivalent and hesitant emotions in video to enhance digital behavioral interventions. It proposes an equally weighted ensemble model built upon frozen multimodal embeddings—integrating facial, audio, textual, and postural features—and incorporates probability calibration to improve generalization. Under strict no-data-leakage conditions, the system achieves a macro F1 score of 0.7358 on the public test set and attains 0.7361 on its first submission to the private test set, demonstrating strong generalization to unseen participants. The study’s novelty lies in its calibrated, equally weighted multimodal fusion strategy and the systematic design of five private test submissions to rigorously evaluate model performance.

Abstract

Ambivalence and hesitancy (A/H) undermine digital behaviour-change interventions, and recognizing them automatically from video is the goal of the ABAW A/H challenge on the BAH dataset. We describe our system for the 11th edition of the challenge: a calibrated, equal-weight ensemble of three fusion models over frozen face, audio, text, and pose embeddings, which reaches 0.7358 macro-F1 on the public test set. This year's private test, released on a disjoint set of 30 new participants, is scored on five allowed submissions; we report the configuration and rationale of each of our five submissions, and, where already available, the private-test score obtained. Our first submission, an exact replica of the calibrated ensemble tuned only on public validation, scored 0.7361 macro-F1 on the private test, matching our public-test estimate almost exactly and confirming the pipeline generalizes to unseen participants without leakage.

Problem

• ambivalence
• hesitancy
• multimodal recognition
• video analysis
• behavior-change interventions

Innovation

• calibrated ensemble
• multimodal fusion
• ambivalence recognition
• hesitancy detection
• generalization to unseen participants