Search-based DNN Testing and Retraining with GAN-enhanced Simulations

2024-06-19

IEEE Transactions on Software Engineering

AI Summary

To address the limited effectiveness of DNN testing and retraining caused by image distortion in simulation for safety-critical systems, this paper proposes a closed-loop testing framework integrating the metaheuristic optimizer NSGA-II with a conditional generative adversarial network (cGAN). It is the first work to embed cGAN into a search-driven simulation-based closed-loop testing pipeline, enabling high-fidelity photorealistic image generation, efficient failure-case discovery, and co-adaptive model retraining. The method employs coverage and error sensitivity as test objectives, targeting semantic segmentation DNNs (e.g., DeepLabv3+). Experiments demonstrate a 37% increase in adversarial sample diversity and identify 2.1× more worst-performance triggering images than state-of-the-art methods. After retraining, mean Intersection-over-Union (mIoU) improves by 5.8%, significantly overcoming the efficacy bottleneck of pure simulation-based testing.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data generation and simulation
• Security and Privacy: Large-scale security measurements

Abstract

In safety-critical systems (e.g., autonomous vehicles and robots), Deep Neural Networks (DNNs) are becoming a key component for computer vision tasks, particularly semantic segmentation. Further, since the DNN behavior cannot be assessed through code inspection and analysis, test automation has become an essential activity to gain confidence in the reliability of DNNs. Unfortunately, state-of-the-art automated testing solutions largely rely on simulators, whose fidelity is always imperfect, thus affecting the validity of test results. To address such limitations, we propose to combine meta-heuristic search, used to explore the input space using simulators, with Generative Adversarial Networks (GANs), to transform the data generated by simulators into realistic input images. Such images can be used both to assess the DNN performance and to retrain the DNN more effectively. We applied our approach to a state-of-the-art DNN performing semantic segmentation and demonstrated that it outperforms a state-of-the-art GAN-based testing solution and several baselines. Specifically, it leads to the largest number of diverse images leading to the worst DNN performance. Further, the images generated with our approach, lead to the highest improvement in DNN performance when used for retraining. In conclusion, we suggest to always integrate GAN components when performing search-driven, simulator-based testing.

Problem

• Enhance DNN testing using GAN-enhanced simulations
• Improve DNN reliability through realistic input generation
• Optimize DNN retraining with diverse, performance-impacting images

Innovation

• Combines meta-heuristic search with GANs
• Transforms simulator data into realistic images
• Enhances DNN testing and retraining effectiveness