EPSM: A Novel Metric to Evaluate the Safety of Environmental Perception in Autonomous Driving

2025-12-17

AI Summary

Existing perception evaluation metrics (e.g., Precision/Recall/F1) emphasize aggregate accuracy while neglecting the safety-critical impact of false positives—particularly hazardous misclassifications that may trigger severe autonomous driving accidents, even in high-scoring systems. To address this fundamental gap, we propose EPSM—the first safety-oriented environmental perception metric that jointly models risk coupling between object detection and lane-line detection. EPSM introduces lightweight object- and lane-safety measures, incorporates risk-sensitive false-positive cost modeling, and performs cross-task safety dependency analysis. Empirical evaluation on the DeepAccident dataset demonstrates that EPSM effectively identifies numerous “high-accuracy, high-risk” false detections, significantly improving detection of catastrophe-inducing errors. It thereby bridges a critical blind spot of conventional accuracy metrics in accident prevention, providing an interpretable, quantifiable theoretical framework and practical tool for safety-driven perception system evaluation.

Technology Category

• Natural Language Processing: Safety and Robustness
• Search and Optimization: Evaluation and Analysis
• Machine Learning: Evaluation and Analysis

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Extensive evaluation of perception systems is crucial for ensuring the safety of intelligent vehicles in complex driving scenarios. Conventional performance metrics such as precision, recall and the F1-score assess the overall detection accuracy, but they do not consider the safety-relevant aspects of perception. Consequently, perception systems that achieve high scores in these metrics may still cause misdetections that could lead to severe accidents. Therefore, it is important to evaluate not only the overall performance of perception systems, but also their safety. We therefore introduce a novel safety metric for jointly evaluating the most critical perception tasks, object and lane detection. Our proposed framework integrates a new, lightweight object safety metric that quantifies the potential risk associated with object detection errors, as well as an lane safety metric including the interdependence between both tasks that can occur in safety evaluation. The resulting combined safety score provides a unified, interpretable measure of perception safety performance. Using the DeepAccident dataset, we demonstrate that our approach identifies safety critical perception errors that conventional performance metrics fail to capture. Our findings emphasize the importance of safety-centric evaluation methods for perception systems in autonomous driving.

Problem

• Develops a new safety metric for autonomous driving perception
• Evaluates object and lane detection errors for potential risks
• Identifies safety-critical errors missed by conventional performance metrics

Innovation

• Introduces a novel safety metric for object and lane detection
• Integrates lightweight object safety metric quantifying detection error risks
• Provides unified interpretable safety score using combined metric framework