Left shifting analysis of Human-Autonomous Team interactions to analyse risks of autonomy in high-stakes AI systems

2025-12-03

AI Summary

In high-risk autonomous systems, AI failures interacting with human operators under stress frequently trigger severe accidents; however, existing approaches lack systematic, early identification of human-AI collaborative failure modes. Method: This paper proposes a “shift-left” human-AI collaborative risk identification framework that integrates human-AI interaction modeling, failure mode analysis, and operational domain coverage assessment into the earliest system design phase—grounded in LaMonica et al.’s human-AI collaboration theory—to enable comprehensive, operation-domain-wide risk characterization. Contribution/Results: Evaluated on an AI assistant in command-and-control scenarios, the framework significantly enhances robustness-aware design of high-risk AI systems, reducing time cost, resource expenditure, and deployment failure risk. Its core innovation lies in the first systematic embedding of human-AI collaborative failure analysis into the front-end of the system lifecycle, thereby enabling trustworthy development of high-assurance AI systems.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Developing high-stakes autonomous systems that include Artificial Intelligence (AI) components is complex; the consequences of errors can be catastrophic, yet it is challenging to plan for all operational cases. In stressful scenarios for the human operator, such as short decision-making timescales, the risk of failures is exacerbated. A lack of understanding of AI failure modes obstructs this and so blocks the robust implementation of applications of AI in smart systems. This prevents early risk identification, leading to increased time, risk and cost of projects. A key tenet of Systems Engineering and acquisition engineering is centred around a"left-shift"in test and evaluation activities to earlier in the system lifecycle, to allow for"accelerated delivery of [systems] that work". We argue it is therefore essential that this shift includes the analysis of AI failure cases as part of the design stages of the system life cycle. Our proposed framework enables the early characterisation of risks emerging from human-autonomy teaming (HAT) in operational contexts. The cornerstone of this is a new analysis of AI failure modes, built on the seminal modelling of human-autonomy teams laid out by LaMonica et al., 2022. Using the analysis of the interactions between human and autonomous systems and exploring the failure modes within each aspect, our approach provides a way to systematically identify human-AI interactions risks across the operational domain of the system of interest. The understanding of the emergent behaviour enables increased robustness of the system, for which the analysis should be undertaken over the whole scope of its operational design domain. This approach is illustrated through an example use case for an AI assistant supporting a Command&Control (C2) System.

Problem

• Identifies risks in human-autonomy teaming within high-stakes AI systems.
• Enables early analysis of AI failure modes during system design stages.
• Systematically assesses human-AI interaction risks across operational domains.

Innovation

• Early AI failure analysis in design stages
• Systematic human-autonomy interaction risk identification
• Framework based on human-autonomy teaming modeling