A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resources

2026-09-05

AI Summary

本文综述了多模态面部状态分析，通过整合视觉、音频等信息解决传统单模态方法的环境敏感性和弱解释性问题，利用多任务学习提升表情识别精度与泛化能力。

Abstract

Facial state analysis plays a crucial role in understanding human expressions, psychological modeling, and human computer interaction. Traditional unimodal vision-based methods are often limited by environmental sensitivity and weak interpretability. Multimodal facial state analysis addresses these issues by integrating complementary cues from visual, audio, textual, physiological, and other related modalities. This survey emphasizes two key aspects: on one hand, multimodal learning enables contextual semantic understanding for improved facial state reasoning and leverages interpretable language generation to enhance model explainability; on the other hand, multi-task learning allows simultaneous analysis of expressions, action units (AUs), and face-based soft biometrics (e.g., age, gender), effectively capturing fine-grained expressions and improving cross-scene generalization. This survey reviews core tasks, representative methods, and datasets in multimodal facial state analysis, focusing on facial expression recognition, AU detection, and face-based soft biometric estimation, and emphasizing the unique value of language in providing contextual semantics, enhancing reasoning, and generating explanations. The survey aims to provide an up-to-date overview of the literature and to highlight future research directions for multimodal, interpretable, and multi-task adaptive facial state analysis.

Problem

• multimodal facial state analysis
• environmental sensitivity
• weak interpretability

Innovation

• multimodal learning
• interpretable language generation
• multi-task learning