End-to-End Facial Expression Detection in Long Videos

2025-04-10

AI Summary

Existing facial micro-expression detection methods typically adopt a two-stage paradigm—separating spotting from recognition—which suffers from error propagation, fragmented feature learning, and the absence of joint optimization. To address these limitations, this paper proposes FEDN, the first end-to-end unified framework for jointly modeling micro-expression spotting and classification in long videos. Its key contributions are: (1) a spatiotemporally sensitive feature extraction module integrating segment-level and sliding-window attention; (2) a joint loss function explicitly designed to optimize spotting and recognition in synergy; and (3) a fully parametric end-to-end training strategy. Evaluated on CASME² and CASME³ benchmarks, FEDN achieves state-of-the-art performance on both spotting and classification tasks, demonstrating substantial improvements in detection accuracy and robustness over prior approaches.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Facial expression detection involves two interrelated tasks: spotting, which identifies the onset and offset of expressions, and recognition, which classifies them into emotional categories. Most existing methods treat these tasks separately using a two-step training pipelines. A spotting model first detects expression intervals. A recognition model then classifies the detected segments. However, this sequential approach leads to error propagation, inefficient feature learning, and suboptimal performance due to the lack of joint optimization of the two tasks. We propose FEDN, an end-to-end Facial Expression Detection Network that jointly optimizes spotting and recognition. Our model introduces a novel attention-based feature extraction module, incorporating segment attention and sliding window attention to improve facial feature learning. By unifying two tasks within a single network, we greatly reduce error propagation and enhance overall performance. Experiments on CASME}^2 and CASME^3 demonstrate state-of-the-art accuracy for both spotting and detection, underscoring the benefits of joint optimization for robust facial expression detection in long videos.

Problem

• Jointly optimizes spotting and recognition for facial expressions
• Reduces error propagation in expression detection and classification
• Improves feature learning with attention-based modules in videos

Innovation

• End-to-end joint optimization for expression tasks
• Attention-based feature extraction with dual attention
• Unified network reduces error propagation significantly