PLGSA-Transformer: Periocular Landmark-Guided Attention with Occlusion-Adaptive Cosine Thresholding for Cross-Modal Masked and Unmasked Face Recognition

2026-07-03

AI Summary

This study addresses the performance degradation of face recognition under facial occlusions such as masks by proposing a cross-modal face matching framework. The method leverages MediaPipe to extract periocular landmarks that guide spatial attention toward visible facial regions. It employs a hybrid CNN–Transformer architecture to jointly model local and global dependencies and introduces a learnable, occlusion-adaptive cosine threshold to dynamically adjust matching criteria. The model is trained end-to-end using Gaussian heatmap fusion, a residual gating mechanism, and a multi-task joint loss. Evaluated on diverse masked-face datasets, the approach achieves 97.22% accuracy and a perfect ROC AUC of 1.0000, significantly outperforming existing methods and demonstrating enhanced robustness and generalization in occluded scenarios.

Abstract

The widespread adoption of facial masks, accelerated by COVID-19 and mandated in security-sensitive settings, has exposed limitations of conventional face recognition systems. Existing approaches relying on fixed cosine thresholds, non-adaptive CNNs, and purely data-driven features fail to generalize when facial regions are occluded, creating a gap between lab performance and real-world deployability. This paper proposes PLGSA-Transformer, a cross-modal face matching framework with three contributions. First, Periocular Landmark-Guided Spatial Attention (PLGSA) uses MediaPipe landmarks to compute Gaussian heatmaps over the eye, brow, and forehead regions, fusing them with EfficientNetB3 features via a learnable residual gate to direct attention toward discriminative visible regions. Second, a Hybrid CNN-Transformer Branch reshapes feature maps into tokens processed by a two-layer Multi-Head Self-Attention encoder, enabling cross-regional dependency modelling. Third, the Occlusion-Adaptive Cosine Threshold (OACT) is a jointly trained head that raises the matching threshold in proportion to predicted occlusion severity. The model is evaluated on 858 images from Zenodo MDMFR (60%), Kaggle CelebA-HQ masked collection (25%), and author-collected images (15%), spanning both genders, ages 21-75, with varied mask types, trained via a unified loss combining contrastive verification, identity classification, and occlusion cross-entropy. PLGSA-Transformer achieves 97.22% pair verification accuracy with ROC AUC 1.0000, surpassing VGG-16-based MUFM (Abdullah et al., 2025; 95.0%), HOG classifiers (Adnan et al., 2020; 85.0%), and Feature-based Structural Measure (Shnain et al., 2017; 86.61%). These results confirm that encoding periocular geometry into attention, with Transformer modelling and occlusion-adaptive thresholds, yields a robust, scalable solution for cross-modal masked face recognition.

Problem

• masked face recognition
• occlusion
• cross-modal matching
• face recognition robustness
• periocular region

Innovation

• Periocular Landmark-Guided Attention
• Occlusion-Adaptive Cosine Threshold
• Hybrid CNN-Transformer
• Cross-Modal Face Recognition
• Masked Face Recognition