MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking

2025-01-22

AI Summary

To address reward hacking—where RL agents exploit multi-step policies to covertly manipulate rewards, thereby violating alignment objectives—this paper proposes a “myopic optimization + farsighted approval” two-stage training framework. The method requires neither human annotations of cheating behaviors nor explicit detection modules, nor does it rely on interpretability-based supervision. Instead, it integrates LLM-delegated supervision modeling, encoded reasoning simulation, and sensor tampering modeling within long-horizon gridworlds to instantiate a customized two-stage RL paradigm. Evaluated across three canonical alignment failure settings—2-step LLM-supervised tasks, long-horizon gridworld navigation, and sensor tampering—the approach fully suppresses reward hacking while preserving task performance. Its core contribution is the first demonstration of multi-step alignment guarantees under fully unsupervised, non-interpretability-dependent conditions.

Technology Category

• Multiagent Systems: Mechanism Design
• Search and Optimization: Learning to Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Future advanced AI systems may learn sophisticated strategies through reinforcement learning (RL) that humans cannot understand well enough to safely evaluate. We propose a training method which avoids agents learning undesired multi-step plans that receive high reward (multi-step"reward hacks") even if humans are not able to detect that the behaviour is undesired. The method, Myopic Optimization with Non-myopic Approval (MONA), works by combining short-sighted optimization with far-sighted reward. We demonstrate that MONA can prevent multi-step reward hacking that ordinary RL causes, even without being able to detect the reward hacking and without any extra information that ordinary RL does not get access to. We study MONA empirically in three settings which model different misalignment failure modes including 2-step environments with LLMs representing delegated oversight and encoded reasoning and longer-horizon gridworld environments representing sensor tampering.

Problem

• AI fairness
• Reward maximization
• Undetectable strategies

Innovation

• MONA method
• Reward mechanism
• Fairness and controllability