A Black Swan Hypothesis: The Role of Human Irrationality in AI Safety

2024-07-25

AI Summary

This paper addresses the ambiguity in defining and understanding high-risk, extremely low-probability “black swan events” in AI safety—particularly their unclear origins and ill-specified nature. We introduce the novel concept of *spatial black swan events*: rare, high-impact failures arising not from temporal discontinuities, but from systematic human cognitive biases in value and probability assessment—under otherwise stable environmental conditions. Departing from traditional time-centric frameworks, we propose the first mathematically rigorous definition system integrating cognitive modeling, behavioral economics, and formal logic—enabling computable classification and modeling of black swan phenomena. Our key contributions are: (1) extending analysis beyond unidimensional time to a *spatial* dimension that explicitly captures bias-driven mechanisms; (2) establishing the first formal black swan definition framework explicitly accounting for human perceptual and cognitive limitations; and (3) providing a foundational theoretical basis for designing intervention algorithms that correct such biases and enhance AI robustness.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Humans and AI: Learning Human Values and Preferences

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Black swan events are statistically rare occurrences that carry extremely high risks. A typical view of defining black swan events is heavily assumed to originate from an unpredictable time-varying environments; however, the community lacks a comprehensive definition of black swan events. To this end, this paper challenges that the standard view is incomplete and claims that high-risk, statistically rare events can also occur in unchanging environments due to human misperception of their value and likelihood, which we call as spatial black swan event. We first carefully categorize black swan events, focusing on spatial black swan events, and mathematically formalize the definition of black swan events. We hope these definitions can pave the way for the development of algorithms to prevent such events by rationally correcting human perception.

Problem

• Defining spatial black swan events in unchanging environments
• Formalizing mathematical definitions for black swan events
• Developing algorithms to correct human misperception risks

Innovation

• Defines spatial black swan events mathematically
• Challenges standard unpredictable event assumptions
• Proposes correcting human perception algorithmically