SPOT-Face: Forensic Face Identification using Attention Guided Optimal Transport

2026-01-14

AI Summary

This work addresses the challenges in forensic identity verification arising from the frequent absence of reference DNA samples and the difficulty of modeling structural correspondences across heterogeneous modalities—such as skull, sketch, and facial photographs. To this end, the authors propose a unified framework based on superpixel graphs, which, for the first time, integrates an attention-guided optimal transport mechanism into forensic face recognition. By leveraging graph neural networks to extract multimodal image embeddings and employing an attention mechanism to precisely align cross-domain structural correspondences, the method achieves robust identity matching. Experimental results on the IIT_Mandi_S2F and CUFS datasets demonstrate that the proposed approach significantly outperforms existing graph-based methods in terms of Recall and mean Average Precision (mAP), thereby enhancing both accuracy and robustness in cross-modal forensic identification.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Person identification in forensic investigations becomes very challenging when common identification means for DNA (i.e., hair strands, soft tissue) are not available. Current methods utilize deep learning methods for face recognition. However, these methods lack effective mechanisms to model cross-domain structural correspondence between two different forensic modalities. In this paper, we introduce a SPOT-Face, a superpixel graph-based framework designed for cross-domain forensic face identification of victims using their skeleton and sketch images. Our unified framework involves constructing a superpixel-based graph from an image and then using different graph neural networks(GNNs) backbones to extract the embeddings of these graphs, while cross-domain correspondence is established through attention-guided optimal transport mechanism. We have evaluated our proposed framework on two publicly available dataset: IIT\_Mandi\_S2F (S2F) and CUFS. Extensive experiments were conducted to evaluate our proposed framework. The experimental results show significant improvement in identification metrics ( i.e., Recall, mAP) over existing graph-based baselines. Furthermore, our framework demonstrates to be highly effective for matching skulls and sketches to faces in forensic investigations.

Problem

• forensic face identification
• cross-domain correspondence
• skull-to-face matching
• sketch-to-face matching
• structural correspondence

Innovation

• superpixel graph
• graph neural networks
• optimal transport
• cross-domain face identification
• attention mechanism