Real-World Perturbation Testing of Autonomous Driving Systems

2026-07-06

AI Summary

This study addresses a critical gap in robustness evaluation for autonomous driving systems by systematically assessing 72 camera and LiDAR perturbations across three levels: model-level, hardware-in-the-loop, and real-vehicle closed-loop testing. Covering both end-to-end vision-based systems and modular LiDAR perception-planning stacks, the work reveals that model-level metrics often fail to reliably predict safety-critical failures in real-world driving. Notably, even subtle camera perturbations can induce significant vehicle control instability, whereas degradation in LiDAR perception shows weaker correlation with system-level failures. These findings underscore the irreplaceable value of real-world, system-level testing and establish a multi-tiered experimental framework for comprehensive robustness assessment in autonomous driving.

Abstract

Autonomous Driving Systems (ADS) must operate reliably under diverse conditions, yet representative data for rare or adverse scenarios is difficult to obtain. Perturbation-based testing is widely used to assess robustness, but most studies focus on offline datasets or simulation, leaving open questions about how such results translate to real-world driving. We present a large-scale study of 72 camera and LiDAR perturbations, evaluated across three testing modalities: offline model-level analysis, hardware-in-the-loop execution, and closed-loop system-level testing on a full-scale autonomous vehicle. The study covers both an end-to-end vision-based driving model and a modular LiDAR-based perception and planning stack. Our results reveal a clear gap between testing levels. For camera-based systems, perturbations with limited offline impact can still induce unstable control and failures in real-world driving. For LiDAR-based systems, degradation is more consistent at the perception level but weakly predictive of system-level failures. Across both modalities, model-level metrics alone are insufficient to identify the most harmful perturbations. We further show that real-time feasibility is a key constraint in real-world testing, and that robustness observations obtained from recorded data do not consistently transfer to closed-loop behavior on a physical vehicle, highlighting the importance of complementary real-world, system-level evaluation.

Problem

• Autonomous Driving Systems
• Perturbation Testing
• Real-World Evaluation
• Robustness
• System-Level Testing

Innovation

• perturbation testing
• autonomous driving systems
• real-world evaluation
• closed-loop testing
• robustness gap