Crafting Physical Adversarial Examples by Combining Differentiable and Physically Based Renders

2025-05-07

AI Summary

To address the insufficient adversarial robustness evaluation of autonomous driving object detectors in the physical world, this paper proposes PAV-Camou—a method for generating physically realizable, robust adversarial camouflage patterns printable on real vehicles. PAV-Camou innovatively integrates differentiable rendering (enabling gradient-based optimization) with physics-based rendering (accurately modeling lighting, material properties, and geometry), while explicitly optimizing the 2D texture-to-3D vehicle mapping to suppress geometric distortion. This ensures strong generalization across viewpoints, illumination conditions, and dynamic road environments. The end-to-end optimized textures are directly deployable via 2D printing. Experiments demonstrate that PAV-Camou significantly reduces vehicle detection rates of mainstream detectors both in digital simulation and physical deployment—maintaining high adversarial efficacy under multi-angle, multi-illumination, and real-world driving scenarios. Thus, PAV-Camou provides a practical, reliable physical-domain testing tool for evaluating the robustness of autonomous driving systems.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Adversarial Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recently we have witnessed progress in hiding road vehicles against object detectors through adversarial camouflage in the digital world. The extension of this technique to the physical world is crucial for testing the robustness of autonomous driving systems. However, existing methods do not show good performances when applied to the physical world. This is partly due to insufficient photorealism in training examples, and lack of proper physical realization methods for camouflage. To generate a robust adversarial camouflage suitable for real vehicles, we propose a novel method called PAV-Camou. We propose to adjust the mapping from the coordinates in the 2D map to those of corresponding 3D model. This process is critical for mitigating texture distortion and ensuring the camouflage's effectiveness when applied in the real world. Then we combine two renderers with different characteristics to obtain adversarial examples that are photorealistic that closely mimic real-world lighting and texture properties. The method ensures that the generated textures remain effective under diverse environmental conditions. Our adversarial camouflage can be optimized and printed in the form of 2D patterns, allowing for direct application on real vehicles. Extensive experiments demonstrated that our proposed method achieved good performance in both the digital world and the physical world.

Problem

• Generating robust adversarial camouflage for real vehicles
• Improving photorealism in training examples for physical application
• Ensuring camouflage effectiveness under diverse environmental conditions

Innovation

• Adjusts 2D to 3D mapping to reduce texture distortion
• Combines differentiable and physically based renderers
• Generates printable 2D adversarial camouflage patterns