Metanormative Theory for RL-Based Moral Agents

2026-08-08

AI Summary

Current reinforcement learning agents lack a unified framework for moral judgment and theoretical guidance, making it difficult to ensure their behaviors align with human values. This work introduces meta-normative theory into the reinforcement learning paradigm for the first time, establishing a design framework for moral agents that integrates philosophical foundations with computational mechanisms. It further proposes an operational taxonomy and evaluation criteria for morally relevant agent behaviors. By clearly defining standards for assessing the morality of reinforcement learning agents, this study not only advances the theoretical grounding for comparing, selecting, and refining value-alignment methods but also facilitates the transition of machine ethics from abstract principles toward quantifiable evaluation.

Abstract

The overlapping disciplines of machine ethics and value alignment are concerned with designing artificial agents that are aligned with human values and that act in ethically acceptable ways. A recent trend in these disciplines is the use of reinforcement learning (RL) to design such agents, sidelining the philosophical literature that used to play a more central role. Against this backdrop, this paper pursues two goals. The first is to draw out ideas from recent work in metanormative theory that can be useful for designing artificial moral and value-aligned agents. The second is to examine the RL architecture through the lens of these ideas. This will give us clearer criteria for when an RL agent's behavior can be classified as moral, as well as a basis for evaluating and comparing different RL-based approaches to machine ethics and value alignment.

Problem

• machine ethics
• value alignment
• reinforcement learning
• metanormative theory
• moral agents

Innovation

• metanormative theory
• reinforcement learning
• value alignment
• machine ethics
• moral agents