A Controlled Study of CLIP-Based Body-Scene Fusion for Emotion Recognition in Context

2026-06-20

AI Summary

This work addresses emotion recognition in natural images by jointly modeling human body posture and scene context. The authors propose a dual-stream architecture that encodes the body region using ResNet-18 and the entire scene using CLIP’s ViT-B/16, followed by fusion to predict 26 discrete emotion categories along with continuous valence, arousal, and dominance dimensions. Comprehensive evaluation reveals that, after incorporating CLIP-based scene representations, various debiasing and few-shot strategies—including causal intervention, CLEF-lite, and ASL tuning—fail to improve performance. The base dual-stream model achieves 34.52% mAP on the EMOTIC test set, outperforming all variants, suggesting that CLIP already captures sufficient scene semantics. Future efforts should therefore focus on modeling label relationships and fine-grained subject-scene interactions.

Abstract

Apparent emotion in natural images is often not visible from the face alone. The face may be small, hidden, or neutral, while posture and scene context carry much of the evidence. This work studies context-aware emotion recognition on EMOTIC with an image-only two-stream model. A ResNet-18 body stream encodes the target-person crop, and a CLIP ViT-B/16 scene stream encodes the full image. The fused feature predicts 26 categorical emotion labels and the continuous valence, arousal, and dominance values. This study examines whether small context-debiasing or rare-class training changes still help after adding a CLIP scene encoder. The clean two-stream model is compared with simplified CCIM-style intervention, CLEF-lite context-bias subtraction, ASL tuning, and class-balanced sampling under the same implementation pipeline. No tested variant improves over the clean two-stream model, which achieves 34.52% mAP on the EMOTIC test split. CLIP gives the model broad scene semantics, but the simplified causal, counterfactual, and rare-class changes do not automatically improve performance. Most remaining errors are in rare and subtle emotion categories, so the next step should focus on label relationships and finer subject-context interaction.

Problem

• emotion recognition
• context-aware
• body-scene fusion
• CLIP
• EMOTIC

Innovation

• CLIP-based fusion
• context-aware emotion recognition
• two-stream architecture
• scene semantics
• rare-class learning