Moral Rebel Agents: Decision-Making Under Conflicting Obligations

2026-09-13

AI Summary

本文探讨了自主代理在执行用户任务时遇到道德义务冲突的问题，通过构建五种代理架构来解决，并在搜索救援场景中评估这些架构的行为表现。

Abstract

Autonomous agents are typically obliged to follow user-assigned tasks. However, strict obedience may conflict with moral obligations that arise during execution. This paper investigates \textbf{moral rebellion}: the ability of an autonomous agent to deviate from a user-assigned task when morally justified. We formalize five agent architectures: an \textit{amoral agent} that pursues assigned tasks without considering moral obligations, and four forms of \textbf{moral rebel agency}: (1) \textit{utilitarian agents} that opportunistically maximize task outcomes; (2) \textit{deontic} agents that enforce normative constraints; (3) \textit{utilitarian-deontic} (UD) agents that combine deontic constraints with utilitarian reasoning; and (4) \textit{dutiful} agents that additionally preserve commitments to assigned tasks. We implement these architectures within a hierarchical task network planning framework and evaluate them in a Mini Search-and-Rescue domain that exposes trade-offs among assigned-task completion, opportunistic rescue, and norm compliance. Our empirical results show that the proposed agents exhibit distinct trade-offs among rescue results, assigned-task completion, and norm compliance. In particular, the preservation of task commitments emerges as an important dimension of moral rebellion, for which the UD and dutiful agents produce substantially different behaviors despite their shared utilitarian and deontological foundations. These findings highlight the importance of commitment-aware moral reasoning for autonomous agents operating in morally consequential environments.

Problem

• moral rebellion
• conflicting obligations
• autonomous agent
• decision-making

Innovation

• moral rebellion
• autonomous agents
• utilitarian-deontic (UD) agents
• dutiful agents
• commitment-aware moral reasoning