From Detection to Correction: Backdoor-Resilient Face Recognition via Vision-Language Trigger Detection and Noise-Based Neutralization

2025-08-07

AI Summary

Facial recognition systems are vulnerable to backdoor attacks, where adversaries embed subtle triggers—e.g., stickers or masks—into a small subset of training images, causing the model to misclassify adversarial users as targeted identities during authentication. Existing defenses struggle to simultaneously achieve high poisoning-sample detection accuracy and preserve data utility. This paper proposes TrueBiometric, the first framework that synergistically integrates multiple vision-language models (VLMs) with a majority-voting mechanism to precisely localize triggers. It further combines gradient-based trigger identification with targeted calibration noise injection to enable lossless repair of poisoned samples. Evaluated across multiple benchmarks, TrueBiometric achieves 100% detection and correction rates for backdoor images while maintaining the original face recognition accuracy—outperforming state-of-the-art defenses by a significant margin.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Biometric systems, such as face recognition systems powered by deep neural networks (DNNs), rely on large and highly sensitive datasets. Backdoor attacks can subvert these systems by manipulating the training process. By inserting a small trigger, such as a sticker, make-up, or patterned mask, into a few training images, an adversary can later present the same trigger during authentication to be falsely recognized as another individual, thereby gaining unauthorized access. Existing defense mechanisms against backdoor attacks still face challenges in precisely identifying and mitigating poisoned images without compromising data utility, which undermines the overall reliability of the system. We propose a novel and generalizable approach, TrueBiometric: Trustworthy Biometrics, which accurately detects poisoned images using a majority voting mechanism leveraging multiple state-of-the-art large vision language models. Once identified, poisoned samples are corrected using targeted and calibrated corrective noise. Our extensive empirical results demonstrate that TrueBiometric detects and corrects poisoned images with 100% accuracy without compromising accuracy on clean images. Compared to existing state-of-the-art approaches, TrueBiometric offers a more practical, accurate, and effective solution for mitigating backdoor attacks in face recognition systems.

Problem

• Detect backdoor attacks in face recognition systems
• Mitigate poisoned images without losing data utility
• Ensure system reliability against unauthorized access

Innovation

• Uses vision-language models for trigger detection
• Applies noise-based neutralization to correct samples
• Ensures 100% accuracy without clean image compromise