FaceInsight: A Multimodal Large Language Model for Face Perception

2025-04-22

AI Summary

Existing general-purpose multimodal large language models (MLLMs) exhibit limitations in facial perception tasks, including inaccurate responses and semantic distortions. To address these issues, we propose Face-MLLM—the first MLLM specifically designed for fine-grained facial understanding. Our approach introduces three key innovations: (i) facial-knowledge-driven vision–language alignment modeling, explicitly encoding deterministic and uncertain dependencies among facial features; (ii) integration of face segmentation maps as a structured auxiliary modality to enhance local perceptual capability; and (iii) support for both zero-shot and fine-tuning evaluation paradigms. Evaluated across three facial perception tasks, Face-MLLM consistently outperforms nine state-of-the-art baseline models under both zero-shot and fine-tuned settings, achieving new SOTA performance. It significantly improves the accuracy and robustness of facial semantic understanding while preserving structural and contextual fidelity.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent advances in multimodal large language models (MLLMs) have demonstrated strong capabilities in understanding general visual content. However, these general-domain MLLMs perform poorly in face perception tasks, often producing inaccurate or misleading responses to face-specific queries. To address this gap, we propose FaceInsight, the versatile face perception MLLM that provides fine-grained facial information. Our approach introduces visual-textual alignment of facial knowledge to model both uncertain dependencies and deterministic relationships among facial information, mitigating the limitations of language-driven reasoning. Additionally, we incorporate face segmentation maps as an auxiliary perceptual modality, enriching the visual input with localized structural cues to enhance semantic understanding. Comprehensive experiments and analyses across three face perception tasks demonstrate that FaceInsight consistently outperforms nine compared MLLMs under both training-free and fine-tuned settings.

Problem

• Improves face perception in multimodal language models
• Addresses inaccurate responses to face-specific queries
• Enhances semantic understanding with face segmentation maps

Innovation

• Visual-textual alignment for facial knowledge
• Face segmentation maps as auxiliary modality
• Fine-grained facial information modeling