Mutual Information Guided Backdoor Mitigation for Pre-Trained Encoders

2024-06-05

IEEE Transactions on Information Forensics and Security

AI Summary

Self-supervised learning (SSL) pre-trained encoders are vulnerable to backdoor attacks, yet existing mitigation methods rely on labeled data—rendering them inapplicable during unsupervised pre-training. This paper proposes the first label-free backdoor mitigation framework: it treats the potentially compromised encoder as a teacher and localizes benign knowledge regions via layer-wise feature mutual information estimation; employs a randomly initialized student network to prevent backdoor inheritance; and jointly optimizes cloning loss and attention loss to disentangle backdoors while preserving semantic representations. The method requires only ≤5% clean pre-training data and achieves significant reductions in attack success rates across two major SSL backdoor attack types after lightweight fine-tuning. Evaluated against seven state-of-the-art baselines, it attains superior performance. Its core innovation lies in a mutual-information-guided selective knowledge distillation mechanism.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Self-supervised learning (SSL) is increasingly attractive for pre-training encoders without requiring labeled data. Downstream tasks built on top of those pre-trained encoders can achieve nearly state-of-the-art performance. The pre-trained encoders by SSL, however, are vulnerable to backdoor attacks as demonstrated by existing studies. Numerous backdoor mitigation techniques are designed for downstream task models. However, their effectiveness is impaired and limited when adapted to pre-trained encoders, due to the lack of label information when pre-training. To address backdoor attacks against pre-trained encoders, in this paper, we innovatively propose a mutual information guided backdoor mitigation technique, named MIMIC(Mutual Information guided backdoor MItigation for pre-trained enCoders). MIMIC uses the potentially backdoored encoder as the teacher network and applies knowledge distillation to create a clean student encoder from it. Different from existing knowledge distillation approaches, MIMIC initializes the student with random weights, inheriting no backdoors from teacher nets. Then MIMIC leverages mutual information between each layer and extracted features to locate where benign knowledge lies in the teacher net, with which distillation is deployed to clone clean features from teacher to student. We craft the distillation loss with two aspects, including clone loss and attention loss, aiming to mitigate backdoors and maintain encoder performance at the same time. Our evaluation conducted on two backdoor attacks in SSL demonstrates that MIMIC can significantly reduce the attack success rate by only utilizing $leq 5$ % of clean pre-training data that is accessible to the defender, surpassing seven state-of-the-art backdoor mitigation techniques. The source code of MIMIC is available at https://github.com/wssun/MIMIC.

Problem

• Mitigating backdoor attacks in self-supervised pre-trained encoders
• Using mutual information to locate and distill clean knowledge
• Reducing attack success rates with minimal clean data

Innovation

• Mutual information guided backdoor mitigation for encoders
• Knowledge distillation with random initialization to avoid backdoors
• Layer-wise mutual information locates benign features for distillation