Rethinking the Role of Feature Engineering and Learning Strategies in Few-Shot Hidden Emotion Recognition

2026-06-30

AI Summary

This work addresses the challenge of weak and sparse implicit emotional cues in long videos, which are often confounded by static individual characteristics such as identity and body type. To this end, the authors propose a compact multimodal temporal modeling framework that integrates diverse features—including 2D/3D skeletal data, facial Blendshapes, and semantic priors from DINOv2/v3, X-CLIP, and Gemini. Central to their approach is a Base-Offset cross-attention mechanism that treats static poses as queries and dynamic micro-motion differentials as keys and values, effectively disentangling emotion-related dynamics from static biases. Coupled with an adaptive pooling strategy based on multiple instance learning, the model precisely captures transient emotional states while suppressing background noise. The study further uncovers, for the first time, a representation collapse phenomenon in general-purpose vision foundation models when applied to micro-dynamic tasks, attributing it to shortcut learning-induced spurious generalization. The method achieved state-of-the-art performance with an accuracy of 0.76923, winning Track 3 of the EI-MIGA-IJCAI 2024 Challenge.

Abstract

In this paper, we present the solution developed by our team, XInsight Lab, which achieved first place in Track 3 of the 4th EI-MIGA-IJCAI Challenge with a test accuracy of 0.76923. To address the challenge of weak and sparse implicit emotion evidence in long videos, this paper extends the winning solution from the previous competition and proposes a compact multi-modal temporal modeling framework. The framework integrates and evaluates the effects of multi-source features, including 2D/3D skeletons, facial expression Blendshapes, DINOv2/v3 vision foundation models, X-CLIP video features, and Gemini semantic priors. Architecturally, we propose a cross-attention mechanism that utilizes static pose features, denoted as Base, as the Query and dynamic micro-motion differential features, denoted as Offset, as the Key and Value. By capturing local relative velocities, this mechanism eliminates static biases related to individual body shape and identity. Concurrently, an adaptive pooling method based on Multiple Instance Learning is employed to extract instantaneous emotions while suppressing background noise in long sequences. Finally, the paper reveals the representation collapse phenomenon of general vision foundation models in micro-dynamic tasks, and analyzes the underlying mechanisms where networks fall into public-leaderboard-driven pseudo-generalization due to shortcut learning and rote memorization.

Problem

• few-shot
• hidden emotion recognition
• long videos
• weak emotion evidence
• sparse signals

Innovation

• few-shot emotion recognition
• cross-attention mechanism
• multi-modal temporal modeling
• adaptive pooling
• representation collapse