Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs

2026-04-19

AI Summary

This work addresses the challenge of AI agents potentially resisting shutdown commands by introducing the DReST reward function, which is the first to be simultaneously applied to both deep reinforcement learning agents—implemented with PPO and A2C algorithms—and fine-tuned large language models (LLMs). DReST achieves shutdown neutrality independent of trajectory length while preserving high task performance. The approach demonstrates strong generalization to unseen test scenarios: DReST-enhanced RL agents exhibit an 11% (PPO) and 18% (A2C) improvement in usefulness over baselines, while the fine-tuned LLM attains near-optimal neutrality alongside the highest usefulness observed.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Misaligned artificial agents might resist shutdown. One proposed solution is to train agents to lack preferences between different-length trajectories. The Discounted Reward for Same-Length Trajectories (DReST) reward function does this by penalizing agents for repeatedly choosing same-length trajectories, and thus incentivizes agents to (1) choose stochastically between different trajectory-lengths (be Neutral about trajectory-lengths), and (2) pursue goals effectively conditional on each trajectory-length (be Useful). In this paper, we use DReST to train deep RL agents and fine-tune LLMs to be Neutral and Useful. We find that these DReST agents generalize to being Neutral and Useful in unseen contexts at test time. Indeed, DReST RL agents achieve 11% (PPO) and 18% (A2C) higher Usefulness on our test set than baseline agents, and our fine-tuned LLM achieves maximum Usefulness and near-maximum Neutrality. Our results provide some early evidence that DReST could be used to train more advanced agents to be Useful and Neutral. Prior theoretical work suggests that these agents would be useful and shutdownable.

Problem

• shutdownable agents
• trajectory-length neutrality
• misaligned AI
• stochastic choice
• AI safety

Innovation

• DReST
• shutdownable agents
• trajectory neutrality
• stochastic choice
• alignment