In-distribution adversarial attacks on object recognition models using gradient-free search

2021-06-30

AI Summary

Neural networks exhibit insufficient robustness to natural perturbations—such as camera pose and illumination variations—within their training distribution, yielding numerous undetected “in-distribution adversarial examples.” This work provides the first systematic empirical confirmation and quantitative characterization of this phenomenon. We propose a gradient-free search method based on the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), enabling efficient identification of failure modes within a parametric rendering distribution. Leveraging controllable synthetic data modeling and a 0.5M unbiased training set, our evaluation reveals in-distribution attack success rates of 71% under camera pose perturbations and 42% under illumination perturbations; these failures generalize to real-world images from ImageNet and Co3D. Our core contributions are: (i) uncovering the intrinsic in-distribution adversarial vulnerability of neural networks to natural variations, and (ii) establishing the first robustness evaluation framework for natural perturbations that requires no external noise, gradient computation, or human annotation.

Abstract

Neural networks are susceptible to small perturbations in the form of 2D rotations and shifts, image crops, and even changes in object colors. Past works attribute these errors to dataset bias, claiming that models fail on these perturbed samples as they do not belong to the training data distribution. Here, we challenge this claim and present evidence of the widespread existence of perturbed images within the training data distribution, which networks fail to classify. We train models on data sampled from parametric distributions, then search inside this data distribution to find such in-distribution adversarial examples. This is done using our gradient-free evolution strategies (ES) based approach which we call CMA-Search. Despite training with a large-scale (0.5 million images), unbiased dataset of camera and light variations, CMA-Search can find a failure inside the data distribution in over 71% cases by perturbing the camera position. With lighting changes, CMA-Search finds misclassifications in 42% cases. These findings also extend to natural images from ImageNet and Co3D datasets. This phenomenon of in-distribution images presents a highly worrisome problem for artificial intelligence -- they bypass the need for a malicious agent to add engineered noise to induce an adversarial attack. All code, datasets, and demos are available at https://github.com/Spandan-Madan/in_distribution_adversarial_examples.

Problem

• Neural Network Robustness
• Image Variability
• Artificial Intelligence Challenge

Innovation

• CMA-Search
• Adversarial Examples
• Neural Network Robustness