Explaining Failures of Cyber-Physical Systems with Actual Causality

2026-06-23

AI Summary

This work addresses the challenge of identifying root causes in autonomous cyber-physical systems—such as self-driving vehicles—when specification violations occur, a task often hindered by insufficient explainability. For the first time, the study introduces theories of actual causality into this domain and proposes two system-agnostic causal explanation algorithms that jointly ensure optimality and computational efficiency. By integrating formal modeling with trajectory analysis from neural network controllers, the approach enables efficient causal inference. Experimental evaluation in an autonomous driving obstacle-avoidance scenario demonstrates that the method accurately pinpoints key causal factors responsible for failures such as collisions, thereby offering effective support for enhancing system trustworthiness and facilitating targeted improvements.

Abstract

Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, face increasingly complex demands, and yet are expected to act reliably. The black-box nature often characterizing such systems, especially those relying on neural components, makes it impossible to fully verify the system behavior prior to deployment. Unfortunately, unexpected failures-when the system does not comply with its specification-are inevitable and may have catastrophic implications. To improve trust in the system and facilitate future mitigation after a failure occurs, it is important to try to derive an explanation for the unexpected system behavior. This paper introduces the novel concept of leveraging the framework of actual causality for CPS failure explanation. Up until now, this framework was only used to derive explanations in the context of simple systems, such as image classifiers. This paper addresses the theoretical gaps and provides the guidance needed to allow for correct explanation derivation in the CPS domain. Beyond the theoretical contribution, the paper presents two novel, practical, system-agnostic explanation derivation algorithms, allowing to prioritize either explanation optimality or derivation efficiency. The approach is demonstrated and evaluated in the context of a neural-network-controlled autonomous car, designed to avoid collisions.

Problem

• Cyber-Physical Systems
• failure explanation
• actual causality
• autonomous systems
• neural components

Innovation

• actual causality
• Cyber-Physical Systems
• failure explanation
• system-agnostic algorithms
• autonomous vehicles