Two Sides of the Same Coin: Learning the Backdoor to Remove the Backdoor

2026-07-06

AI Summary

This work addresses the limitations of existing backdoor defense methods, which often struggle to effectively identify benign samples and remove backdoors implanted via data poisoning during training. To overcome this challenge, the authors propose HARVEY, a novel approach that instead learns an easily trainable backdoor reference model to accurately distinguish poisoned samples. HARVEY integrates a loss-difference-based sample separation strategy with an anti-backdoor learning framework, substantially improving the precision of poisoned sample identification. Extensive experiments demonstrate that HARVEY consistently outperforms state-of-the-art defenses across diverse attack types, datasets, and model architectures, achieving near-perfect backdoor removal—reducing attack success rates to nearly zero—while preserving the model’s natural accuracy with minimal degradation.

Abstract

The community has recently developed various training-time defenses to counter neural backdoors introduced through data poisoning. In light of the observation that a model learns poisonous samples responsible for the backdoor easier than benign samples, these approaches either use a fixed threshold of the training loss for splitting or iteratively learn a reference model as an oracle for identifying benign samples. In particular, the latter has proven effective for anti-backdoor learning. Our method, HARVEY, leverages a similar yet crucially different technique: learning an oracle for poisonous rather than benign samples. Learning a backdoored reference model is significantly easier than learning a reference model on benign data. Consequently, we can identify poisonous samples much more accurately than related work identifies benign samples. This crucial difference enables near-perfect backdoor removal as we demonstrate in our evaluation. HARVEY substantially outperforms related approaches across attack types, datasets, and architectures, lowering the attack success rate to the very minimum at a negligible loss in natural accuracy. The figure below shows an overview of our methods working principle.

Problem

• neural backdoors
• data poisoning
• backdoor removal
• training-time defenses

Innovation

• backdoor defense
• poisonous sample identification
• reference model
• anti-backdoor learning
• data poisoning