Perception Characteristics Distance: Measuring Stability and Robustness of Perception System in Dynamic Conditions under a Certain Decision Rule

2025-06-10

AI Summary

Conventional static evaluation metrics fail to capture the time-varying confidence characteristics of autonomous driving perception, which is significantly affected by dynamic factors such as distance, scene dynamics, and weather. Method: We propose Perception Characteristic Distance (PCD), a novel reliability metric for dynamic environments, defined as the maximum detection distance at which a target remains reliably detectable under a given decision rule—jointly modeling model uncertainty and environmental variability. We further introduce distance-dependent confidence changepoint analysis to formulate the distribution-aware metric mPCD. Additionally, we release SensorRainFall, the first multi-sensor dataset with precise ground-truth distance annotations for paired rainy/sunny conditions. Our end-to-end PCD computation integrates camera, radar, and LiDAR fusion with statistical changepoint detection and uncertainty modeling. Results: Experiments demonstrate that mPCD effectively discriminates perceptual reliability across weather conditions, whereas traditional metrics fail. Code and dataset are publicly available.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

The performance of perception systems in autonomous driving systems (ADS) is strongly influenced by object distance, scene dynamics, and environmental conditions such as weather. AI-based perception outputs are inherently stochastic, with variability driven by these external factors, while traditional evaluation metrics remain static and event-independent, failing to capture fluctuations in confidence over time. In this work, we introduce the Perception Characteristics Distance (PCD) -- a novel evaluation metric that quantifies the farthest distance at which an object can be reliably detected, incorporating uncertainty in model outputs. To support this, we present the SensorRainFall dataset, collected on the Virginia Smart Road using a sensor-equipped vehicle (cameras, radar, LiDAR) under controlled daylight-clear and daylight-rain scenarios, with precise ground-truth distances to the target objects. Statistical analysis reveals the presence of change points in the variance of detection confidence score with distance. By averaging the PCD values across a range of detection quality thresholds and probabilistic thresholds, we compute the mean PCD (mPCD), which captures the overall perception characteristics of a system with respect to detection distance. Applying state-of-the-art perception models shows that mPCD captures meaningful reliability differences under varying weather conditions -- differences that static metrics overlook. PCD provides a principled, distribution-aware measure of perception performance, supporting safer and more robust ADS operation, while the SensorRainFall dataset offers a valuable benchmark for evaluation. The SensorRainFall dataset is publicly available at https://www.kaggle.com/datasets/datadrivenwheels/sensorrainfall, and the evaluation code is open-sourced at https://github.com/datadrivenwheels/PCD_Python.

Problem

• Measure perception system stability under dynamic conditions
• Address variability in AI-based perception outputs
• Evaluate detection reliability across different weather conditions

Innovation

• Introduces Perception Characteristics Distance (PCD) metric
• Uses SensorRainFall dataset for controlled scenarios
• Applies mPCD to capture perception reliability differences