Criticality Metrics for Relevance Classification in Safety Evaluation of Object Detection in Automated Driving

2025-12-17

AI Summary

In autonomous driving safety evaluation, distinguishing safety-critical (i.e., “critical”) objects from non-critical ones remains challenging due to the semantic gap in conventional perception metrics. Method: This paper proposes the first criticality (relevance) classification framework tailored for object detection systems. It formally defines and models criticality as a quantifiable property, introduces a bidirectional criticality scoring mechanism, and integrates multiple safety-oriented metrics to achieve robust classification. Contribution/Results: Evaluated on the DeepAccident dataset, the framework demonstrates substantial improvements in criticality identification—achieving up to 100% relative accuracy gain over baseline methods. It effectively bridges the semantic deficiency of traditional accuracy-based metrics (e.g., mAP) in safety assessment, offering a novel, interpretable, and quantifiable paradigm for verifying perception system reliability.

Technology Category

• Computer Vision: Object Detection & Categorization
• Natural Language Processing: Safety and Robustness
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Ensuring safety is the primary objective of automated driving, which necessitates a comprehensive and accurate perception of the environment. While numerous performance evaluation metrics exist for assessing perception capabilities, incorporating safety-specific metrics is essential to reliably evaluate object detection systems. A key component for safety evaluation is the ability to distinguish between relevant and non-relevant objects - a challenge addressed by criticality or relevance metrics. This paper presents the first in-depth analysis of criticality metrics for safety evaluation of object detection systems. Through a comprehensive review of existing literature, we identify and assess a range of applicable metrics. Their effectiveness is empirically validated using the DeepAccident dataset, which features a variety of safety-critical scenarios. To enhance evaluation accuracy, we propose two novel application strategies: bidirectional criticality rating and multi-metric aggregation. Our approach demonstrates up to a 100% improvement in terms of criticality classification accuracy, highlighting its potential to significantly advance the safety evaluation of object detection systems in automated vehicles.

Problem

• Analyzing criticality metrics for object detection safety evaluation
• Identifying and assessing relevance classification metrics in automated driving
• Proposing strategies to improve criticality classification accuracy in perception systems

Innovation

• Bidirectional criticality rating for relevance classification
• Multi-metric aggregation to enhance evaluation accuracy
• Empirical validation using DeepAccident safety-critical dataset