CLIP-HandID: Vision-Language Model for Hand-Based Person Identification

2025-06-14

AI Summary

Addressing the challenge of identity recognition from hand-only images—common in forensic investigations such as sexual assault cases—this paper proposes the first CLIP-based cross-modal hand identity recognition framework. Methodologically, it introduces a textual inversion mechanism to learn hand-specific pseudo-tokens, enabling semantic-guided representation learning without textual annotations; it further enhances visual–semantic alignment through contrastive learning and prompt engineering. Evaluated on two large-scale, multi-ethnic hand image datasets, the method significantly outperforms existing approaches, demonstrating strong generalization and high identification accuracy under few-shot and zero-text-label settings. Its interpretability, minimal annotation dependency, and robust performance establish a novel paradigm for forensic image analysis.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

This paper introduces a new approach to person identification based on hand images, designed specifically for criminal investigations. The method is particularly valuable in serious crimes like sexual abuse, where hand images are often the sole identifiable evidence available. Our proposed method, CLIP-HandID, leverages pre-trained foundational vision-language model, particularly CLIP, to efficiently learn discriminative deep feature representations from hand images given as input to the image encoder of CLIP using textual prompts as semantic guidance. We propose to learn pseudo-tokens that represent specific visual contexts or appearance attributes using textual inversion network since labels of hand images are indexes instead text descriptions. The learned pseudo-tokens are incorporated into textual prompts which are given as input to the text encoder of the CLIP to leverage its multi-modal reasoning to enhance its generalization for identification. Through extensive evaluations on two large, publicly available hand datasets with multi-ethnic representation, we show that our method substantially surpasses existing approaches.

Problem

• Identify persons using hand images for criminal investigations
• Leverage CLIP model for hand-based feature representation
• Enhance identification accuracy with multi-ethnic hand datasets

Innovation

• Leverages CLIP for hand image identification
• Uses textual inversion for pseudo-token learning
• Enhances generalization via multi-modal reasoning