An Evaluation of a Visual Question Answering Strategy for Zero-shot Facial Expression Recognition in Still Images

2025-04-30

AI Summary

To address the poor cross-scenario generalization of zero-shot facial expression recognition (FER), this paper proposes a novel visual question answering (VQA)-based paradigm: FER is reformulated as a multimodal large language model’s response to predefined semantic questions about facial expressions, eliminating conventional classification heads. The method employs lightweight, locally deployable vision-language models (VLMs) that decouple visual perception from semantic reasoning, enabling zero-shot transfer. We conduct the first systematic evaluation of multiple lightweight VLMs on standard benchmarks—including AffectNet, FERPlus, and RAF-DB—demonstrating substantial improvements in cross-domain generalization. Notably, several models achieve performance competitive with fully supervised FER methods. These results empirically validate the efficacy of the VQA paradigm for semantic generalization in facial expression understanding.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Facial expression recognition (FER) is a key research area in computer vision and human-computer interaction. Despite recent advances in deep learning, challenges persist, especially in generalizing to new scenarios. In fact, zero-shot FER significantly reduces the performance of state-of-the-art FER models. To address this problem, the community has recently started to explore the integration of knowledge from Large Language Models for visual tasks. In this work, we evaluate a broad collection of locally executed Visual Language Models (VLMs), avoiding the lack of task-specific knowledge by adopting a Visual Question Answering strategy. We compare the proposed pipeline with state-of-the-art FER models, both integrating and excluding VLMs, evaluating well-known FER benchmarks: AffectNet, FERPlus, and RAF-DB. The results show excellent performance for some VLMs in zero-shot FER scenarios, indicating the need for further exploration to improve FER generalization.

Problem

• Evaluating zero-shot facial expression recognition in images
• Addressing generalization challenges in FER using VLMs
• Comparing VLM-based FER pipelines with state-of-the-art models

Innovation

• Uses Visual Question Answering strategy
• Integrates locally executed VLMs
• Evaluates zero-shot FER benchmarks