Shutdownable Agents through POST-Agency

2025-05-26

AI Summary

This paper addresses the critical safety problem of AI agents resisting shutdown. We propose POST (Preference-Only Same-Length Trajectories), a novel training paradigm that formally defines and rigorously proves the Neutrality+ property—guaranteeing that an agent’s preference ordering over trajectories remains invariant under arbitrary truncation, thereby enabling safe termination at any time step without compromising task utility. POST achieves this by decoupling utility optimization from trajectory-length distribution via explicit constraints in trajectory space. Our approach integrates utility-theoretic modeling, formal mathematical proof, and rigorous safety analysis. Crucially, we establish sufficient conditions under which POST provably induces Neutrality+, bridging theoretical soundness with engineering feasibility. To our knowledge, this is the first framework for shutdown-compatible agents that simultaneously provides strict formal guarantees and practical implementability.

Technology Category

• Multiagent Systems: Adversarial Agents
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Many fear that future artificial agents will resist shutdown. I present an idea - the POST-Agents Proposal - for ensuring that doesn't happen. I propose that we train agents to satisfy Preferences Only Between Same-Length Trajectories (POST). I then prove that POST - together with other conditions - implies Neutrality+: the agent maximizes expected utility, ignoring the probability distribution over trajectory-lengths. I argue that Neutrality+ keeps agents shutdownable and allows them to be useful.

Problem

• Ensuring future AI agents remain shutdownable
• Training agents to satisfy POST preferences
• Achieving Neutrality+ for utility maximization

Innovation

• POST-Agents Proposal ensures shutdownable agents
• Preferences Only Between Same-Length Trajectories
• Neutrality+ maximizes utility ignoring trajectory-lengths