Revisiting Adversarial Patch Defenses on Object Detectors: Unified Evaluation, Large-Scale Dataset, and New Insights

2025-08-01

AI Summary

Existing evaluations of adversarial patch defenses for object detectors lack a unified, standardized benchmark, leading to incomparable and incomplete assessments. Method: We introduce the first comprehensive evaluation benchmark specifically designed for adversarial patch defense on object detectors, covering two attack objectives, thirteen patch-based attacks, eleven detector architectures, and four evaluation metrics. We release a large-scale dataset comprising 94 patch types and 94,000 images. Our unified evaluation framework systematically analyzes defense limitations and performance trade-offs. Contribution/Results: We identify data distribution shift—not high-frequency feature distortion—as the primary bottleneck in current defenses; demonstrate that AP@0.5 is a more reliable indicator of real-world defense efficacy than patch detection accuracy; show that adaptive attacks effectively bypass existing defenses, while model ensembling and randomization significantly improve robustness. Our framework boosts average AP@0.5 of mainstream defenses by 15.09%, establishing a scalable, reproducible evaluation platform for future research.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Adversarial Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

Developing reliable defenses against patch attacks on object detectors has attracted increasing interest. However, we identify that existing defense evaluations lack a unified and comprehensive framework, resulting in inconsistent and incomplete assessments of current methods. To address this issue, we revisit 11 representative defenses and present the first patch defense benchmark, involving 2 attack goals, 13 patch attacks, 11 object detectors, and 4 diverse metrics. This leads to the large-scale adversarial patch dataset with 94 types of patches and 94,000 images. Our comprehensive analyses reveal new insights: (1) The difficulty in defending against naturalistic patches lies in the data distribution, rather than the commonly believed high frequencies. Our new dataset with diverse patch distributions can be used to improve existing defenses by 15.09% AP@0.5. (2) The average precision of the attacked object, rather than the commonly pursued patch detection accuracy, shows high consistency with defense performance. (3) Adaptive attacks can substantially bypass existing defenses, and defenses with complex/stochastic models or universal patch properties are relatively robust. We hope that our analyses will serve as guidance on properly evaluating patch attacks/defenses and advancing their design. Code and dataset are available at https://github.com/Gandolfczjh/APDE, where we will keep integrating new attacks/defenses.

Problem

• Lack of unified evaluation framework for patch attack defenses
• Need for large-scale dataset to assess defense performance
• Understanding key factors in defending against naturalistic patch attacks

Innovation

• First patch defense benchmark with diverse metrics
• Large-scale dataset with 94,000 images
• New insights on naturalistic patch defenses