Evaluating Multimodal Large Language Models for Heterogeneous Face Recognition

2026-01-21

AI Summary

This study presents the first systematic evaluation of multimodal large language models (MLLMs) within a rigorous biometric assessment framework for heterogeneous face recognition, specifically examining their performance across cross-spectral scenarios including visible-to-near-infrared (VIS-NIR), visible-to-short-wave infrared (VIS-SWIR), and visible-to-thermal (VIS-THERMAL) imaging. Using standard biometric metrics—such as Equal Error Rate (EER), True Acceptance Rate (TAR), and Acquire Rate—the results demonstrate that current MLLMs significantly underperform compared to specialized face recognition systems, particularly under challenging cross-modal conditions. The findings reveal critical limitations of MLLMs in cross-spectral face matching and establish a foundational benchmark for future research on integrating multimodal foundation models into biometric applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance on a wide range of vision-language tasks, raising interest in their potential use for biometric applications. In this paper, we conduct a systematic evaluation of state-of-the-art MLLMs for heterogeneous face recognition (HFR), where enrollment and probe images are from different sensing modalities, including visual (VIS), near infrared (NIR), short-wave infrared (SWIR), and thermal camera. We benchmark multiple open-source MLLMs across several cross-modality scenarios, including VIS-NIR, VIS-SWIR, and VIS-THERMAL face recognition. The recognition performance of MLLMs is evaluated using biometric protocols and based on different metrics, including Acquire Rate, Equal Error Rate (EER), and True Accept Rate (TAR). Our results reveal substantial performance gaps between MLLMs and classical face recognition systems, particularly under challenging cross-spectral conditions, in spite of recent advances in MLLMs. Our findings highlight the limitations of current MLLMs for HFR and also the importance of rigorous biometric evaluation when considering their deployment in face recognition systems.

Problem

• Multimodal Large Language Models
• Heterogeneous Face Recognition
• Cross-modal Biometrics
• Face Recognition Evaluation

Innovation

• Multimodal Large Language Models
• Heterogeneous Face Recognition
• Cross-spectral Recognition
• Biometric Evaluation
• Infrared Imaging