FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs

2024-07-02

ACM Multimedia

AI Summary

To address two key bottlenecks in Dynamic Facial Expression Recognition (DFER)—inadequate facial dynamic modeling and ambiguous expression semantics—this paper proposes a multimodal fine-grained CLIP framework. Methodologically, it introduces (i) a novel three-level semantic modeling scheme integrating frame-level, segment/keypoint-level, and MLLM-generated descriptive visual–language representations; (ii) a cross-modal contrastive learning mechanism supervised by positive/negative textual labels; and (iii) the first adaptation of AdaptERs to CLIP for parameter-efficient fine-tuning, enabling both zero-shot and few-shot recognition. Evaluated on DFEW, FERV39k, and MAFW, the method achieves state-of-the-art performance in both fully supervised and zero-shot settings, using fewer than 0.5% trainable parameters. It significantly enhances dynamic semantic alignment and generalization across domains and data regimes.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Dynamic Facial Expression Recognition (DFER) is crucial for understanding human behavior. However, current methods exhibit limited performance mainly due to the insufficient utilization of facial dynamics, and the ambiguity of expression semantics, etc. To this end, we propose a novel framework, named Multi-modal Fine-grained CLIP for DFER with AdaptERs (FineCLIPER), incorporating the following novel designs: 1) To better distinguish between similar facial expressions, we extend the class labels to textual descriptions from both positive and negative aspects, and obtain supervision by calculating the cross-modal similarity based on the CLIP model; 2) Our FineCLIPER adopts a hierarchical manner to effectively mine useful cues from DFE videos. Specifically, besides directly embedding video frames as input (low semantic level), we propose to extract the face segmentation masks and landmarks based on each frame (middle semantic level) and utilize the Multi-modal Large Language Model (MLLM) to further generate detailed descriptions of facial changes across frames with designed prompts (high semantic level). Additionally, we also adopt Parameter-Efficient Fine-Tuning (PEFT) to enable efficient adaptation of large pre-trained models (i.e., CLIP) for this task. Our FineCLIPER achieves SOTA performance on the DFEW, FERV39k, and MAFW datasets in both supervised and zero-shot settings with few tunable parameters. Project page: https://haroldchen19.github.io/FineCLIPER-Page/

Problem

• Limited performance in dynamic facial expression recognition
• Scarcity of high-quality data and ambiguous expression semantics
• Insufficient utilization of facial dynamics in current methods

Innovation

• Extends labels to textual descriptions for cross-modal supervision
• Uses hierarchical cues from videos with MLLM descriptions
• Applies Parameter-Efficient Fine-Tuning for CLIP adaptation