ActFER: Agentic Facial Expression Recognition via Active Tool-Augmented Visual Reasoning

2026-04-10

AI Summary

This work proposes ActFER, the first active perception agent framework tailored for facial expression recognition (FER), addressing the limitations of existing multimodal large language model (MLLM)-based approaches that rely on passive perception and struggle to actively acquire and reason about critical visual evidence. ActFER reformulates FER as a dynamic process of tool invocation and multimodal reasoning, leveraging face detection and alignment to focus on informative local regions and integrating action units with emotion labels to enable chain-of-thought visual reasoning. To support adaptive observation decisions and multi-level verifiable rewards, the authors introduce a utility-calibrated GRPO reinforcement learning algorithm (UC-GRPO). Experiments demonstrate that ActFER significantly outperforms current passive MLLM-FER methods across multiple benchmarks, substantially improving both action unit prediction accuracy and overall expression comprehension.

Technology Category

• Machine Learning: Active Learning
• Computer Vision: Video Understanding & Activity Analysis
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have created new opportunities for facial expression recognition (FER), moving it beyond pure label prediction toward reasoning-based affect understanding. However, existing MLLM-based FER methods still follow a passive paradigm: they rely on externally prepared facial inputs and perform single-pass reasoning over fixed visual evidence, without the capability for active facial perception. To address this limitation, we propose ActFER, an agentic framework that reformulates FER as active visual evidence acquisition followed by multimodal reasoning. Specifically, ActFER dynamically invokes tools for face detection and alignment, selectively zooms into informative local regions, and reasons over facial Action Units (AUs) and emotions through a visual Chain-of-Thought. To realize such behavior, we further develop Utility-Calibrated GRPO (UC-GRPO), a reinforcement learning algorithm tailored to agentic FER. UC-GRPO uses AU-grounded multi-level verifiable rewards to densify supervision, query-conditional contrastive utility estimation to enable sample-aware dynamic credit assignment for local inspection, and emotion-aware EMA calibration to reduce noisy utility estimates while capturing emotion-wise inspection tendencies. This algorithm enables ActFER to learn both when local inspection is beneficial and how to reason over the acquired evidence. Comprehensive experiments show that ActFER trained with UC-GRPO consistently outperforms passive MLLM-based FER baselines and substantially improves AU prediction accuracy.

Problem

• Facial Expression Recognition
• Multimodal Large Language Models
• Active Perception
• Visual Reasoning
• Action Units

Innovation

• Agentic FER
• Active Visual Reasoning
• Tool-Augmented MLLM
• Utility-Calibrated GRPO
• Action Unit Grounding