A pragmatic classification of AI incident trajectories

2026-04-23

AI Summary

Existing AI incident databases struggle to disentangle reporting biases, increases in deployment scale, and genuine changes in hazard rates per unit of exposure, leading to substantial uncertainty in risk assessments. This work proposes an innovative framework that, for the first time, decouples exposure from hazard rates by integrating a structured monitoring taxonomy (SORT), a hierarchical calibration estimation pipeline, and large language model–assisted incident matching to enable interpretable categorization of AI incident trajectories. Validated across multiple scenarios, the approach reveals critical limitations in current monitoring systems and provides policymakers with clear evidentiary boundaries and a principled classification basis, thereby advancing a paradigm for exposure-adjusted AI risk governance.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Public AI incident database counts conflate changes in reporting propensity, deployment growth, and shifts in harm frequency per unit of exposure. These issues introduce significant uncertainties challenging public and corporate policy frameworks centred on realized risks. We propose a simple framework that establishes clear points of inquiry, separately estimates exposure from harm-rate trends, and then classifies into meaningful trajectory categories for governance decisions. The framework combines a structured monitoring question format (SORT) to clarify coverage decisions, a tiered estimation procedure calibrated to available evidence, and LLM-assisted incident matching against public databases. Applied to various monitoring questions, we draw conclusions regarding the monitoring ecosystem more broadly: Providing an essential interpretative classification, determining what can and cannot be claimed, and establishing that exposure estimation is required as AI deployments become increasingly common.

Problem

• AI incident
• risk assessment
• exposure estimation
• harm frequency
• policy framework

Innovation

• AI incident classification
• exposure estimation
• harm-rate trends
• SORT framework
• LLM-assisted matching