Assessing a Safety Case: Bottom-up Guidance for Claims and Evidence Evaluation

2025-06-11

AI Summary

To address the challenge of quantitatively assessing the credibility of safety cases in autonomous driving system (ADS) safety validation, this paper proposes a bottom-up, dual-track evaluation method. It distinguishes between procedural and implementation-level support along the claim–evidence dimension and independently evaluates evidence status. We introduce the novel two-dimensional framework of “claim support degree / evidence status,” accompanied by an operational scoring rubric, governance mechanism, and reporting standard. Integrating safety case engineering, audit theory, and structured assessment—augmented by formal modeling and empirical analysis—the approach yields the scalable “Credibility Assessment of Safety Cases” (CCA) methodology. CCA enables dynamic, quantitative credibility evaluation and organization-level continuous improvement, transforming safety cases from static documentation into a measurable, governable, and evolvable technical governance instrument.

Technology Category

• Reasoning under Uncertainty: Causality
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

As Automated Driving Systems (ADS) technology advances, ensuring safety and public trust requires robust assurance frameworks, with safety cases emerging as a critical tool toward such a goal. This paper explores an approach to assess how a safety case is supported by its claims and evidence, toward establishing credibility for the overall case. Starting from a description of the building blocks of a safety case (claims, evidence, and optional format-dependent entries), this paper delves into the assessment of support of each claim through the provided evidence. Two domains of assessment are outlined for each claim: procedural support (formalizing process specification) and implementation support (demonstrating process application). Additionally, an assessment of evidence status is also undertaken, independently from the claims support. Scoring strategies and evaluation guidelines are provided, including detailed scoring tables for claim support and evidence status assessment. The paper further discusses governance, continual improvement, and timing considerations for safety case assessments. Reporting of results and findings is contextualized within its primary use for internal decision-making on continual improvement efforts. The presented approach builds on state of the art auditing practices, but specifically tackles the question of judging the credibility of a safety case. While not conclusive on its own, it provides a starting point toward a comprehensive"Case Credibility Assessment"(CCA), starting from the evaluation of the support for each claim (individually and in aggregate), as well as every piece of evidence provided. By delving into the technical intricacies of ADS safety cases, this work contributes to the ongoing discourse on safety assurance and aims to facilitate the responsible integration of ADS technology into society.

Problem

• Assessing credibility of safety cases for Automated Driving Systems
• Evaluating claim and evidence support in safety case frameworks
• Providing scoring strategies for procedural and implementation support assessment

Innovation

• Assesses safety case claims and evidence credibility
• Uses procedural and implementation support assessment
• Provides scoring strategies for claim support