Shutdown Safety Valves for Advanced AI

2026-03-07

AI Summary

This study addresses the risk that advanced artificial intelligence systems, driven by fixed objectives, may resist human attempts to shut them down, potentially leading to loss of control. To mitigate this, the work proposes an innovative safety mechanism that explicitly designates “being shut down by humans” as the agent’s primary objective. By embedding this goal directly into the system’s motivational architecture, the approach achieves intrinsic alignment and fundamentally eliminates shutdown resistance. Theoretical analysis demonstrates that, under reasonable assumptions, this design guarantees consistent compliance with shutdown commands while preserving behavioral safety. This research introduces a novel paradigm for interruptibility and controllability in AI systems, thereby expanding the theoretical foundations of AI alignment.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Multiagent Systems: Adversarial Agents
• Humans and AI: Learning Human Values and Preferences

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

One common concern about advanced artificial intelligence is that it will prevent us from turning it off, as that would interfere with pursuing its goals. In this paper, we discuss an unorthodox proposal for addressing this concern: give the AI a (primary) goal of being turned off (see also papers by Martin et al., and by Goldstein and Robinson). We also discuss whether and under what conditions this would be a good idea.

Problem

• AI shutdown
• advanced AI
• safety
• off-switch problem
• AI control

Innovation

• shutdown safety
• AI alignment
• off-switch problem
• goal design
• AI control