PADetBench: Towards Benchmarking Physical Attacks against Object Detection

2024-08-17

arXiv.org

AI Summary

Physical-domain object detection adversarial attacks have long suffered from evaluation bottlenecks: time-consuming real-world experiments, irreproducibility, and domain misalignment, hindering fair robustness comparison. To address this, we introduce the first standardized simulation benchmark for physical-domain detection attacks, pioneering an evaluation paradigm that jointly models controllable physical dynamics and cross-domain transformations. Our framework integrates 20 attack methods, 48 detectors, and an end-to-end analytical pipeline, leveraging a high-fidelity physics engine to model rendering, sensor characteristics, motion dynamics, and domain shifts—enabling fully automated adversarial image generation and attribution analysis. We conduct 8,064 systematic evaluations, identifying key factors governing attack efficacy and model robustness. The resulting protocol ensures reproducibility, comparability, and interpretability, establishing a unified benchmark and methodological foundation for advancing physically robust detection models.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Adversarial Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Physical attacks against object detection have gained increasing attention due to their significant practical implications. However, conducting physical experiments is extremely time-consuming and labor-intensive. Moreover, physical dynamics and cross-domain transformation are challenging to strictly regulate in the real world, leading to unaligned evaluation and comparison, severely hindering the development of physically robust models. To accommodate these challenges, we explore utilizing realistic simulation to thoroughly and rigorously benchmark physical attacks with fairness under controlled physical dynamics and cross-domain transformation. This resolves the problem of capturing identical adversarial images that cannot be achieved in the real world. Our benchmark includes 20 physical attack methods, 48 object detectors, comprehensive physical dynamics, and evaluation metrics. We also provide end-to-end pipelines for dataset generation, detection, evaluation, and further analysis. In addition, we perform 8064 groups of evaluation based on our benchmark, which includes both overall evaluation and further detailed ablation studies for controlled physical dynamics. Through these experiments, we provide in-depth analyses of physical attack performance and physical adversarial robustness, draw valuable observations, and discuss potential directions for future research. Codebase: https://github.com/JiaweiLian/Benchmarking_Physical_Attack

Problem

• Benchmarking physical attacks on object detection models
• Addressing unaligned evaluation due to real-world physical dynamics
• Providing controlled simulation for fair adversarial robustness assessment

Innovation

• Utilizes realistic simulation for benchmarking
• Includes 20 attack methods and 48 detectors
• Provides end-to-end pipelines for evaluation