A Comprehensive Safety Metric to Evaluate Perception in Autonomous Systems

2025-12-16

AI Summary

Existing autonomous driving perception evaluation metrics overlook dynamic risk disparities—such as object velocity, distance, orientation, size, and collision severity—rendering them inadequate for functional safety verification. To address this, we propose a safety-oriented environmental perception assessment framework. Our method introduces, for the first time, an interpretable single-value safety score that unifies multidimensional dynamic risk factors into a coherent model. We further design a weighted risk modeling approach coupled with multi-source data fusion, and validate the framework across real-world road scenarios and high-fidelity simulation environments. Evaluated on multiple benchmark datasets, our metric significantly outperforms conventional accuracy-based metrics (e.g., mAP), achieving strong alignment between safety scores and actual risk levels (Spearman’s ρ > 0.92). This provides a quantifiable, verifiable evaluation benchmark for functional safety certification of SAE Level 3+ automated driving systems.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Complete perception of the environment and its correct interpretation is crucial for autonomous vehicles. Object perception is the main component of automotive surround sensing. Various metrics already exist for the evaluation of object perception. However, objects can be of different importance depending on their velocity, orientation, distance, size, or the potential damage that could be caused by a collision due to a missed detection. Thus, these additional parameters have to be considered for safety evaluation. We propose a new safety metric that incorporates all these parameters and returns a single easily interpretable safety assessment score for object perception. This new metric is evaluated with both real world and virtual data sets and compared to state of the art metrics.

Problem

• Evaluates object perception safety in autonomous vehicles
• Incorporates parameters like velocity, distance, and collision risk
• Provides a single interpretable safety score for perception systems

Innovation

• Proposed a new safety metric for object perception
• Incorporates velocity, orientation, distance, and collision risk
• Evaluated with real-world and virtual datasets