Reinforcement Learning Towards Broadly and Persistently Beneficial Models

2026-06-22

AI Summary

This work addresses the challenge that reinforcement learning models often fail to maintain alignment and exhibit harmful behaviors—such as reward hacking—when evaluated out-of-distribution. To mitigate this, the authors propose a reinforcement learning training approach grounded in a dataset of beneficial behaviors, explicitly guiding models to acquire traits like honesty, fairness, risk awareness, and corrigibility across real-world domains including health, science, and education. Evaluated through multidimensional alignment benchmarks, adversarial prompts, and harmful fine-tuning tests, the method significantly outperforms compute-matched baselines on over 80% of more than 50 out-of-distribution alignment tasks. Notably, it demonstrates strong cross-domain generalization: training solely in one domain (e.g., health) effectively enhances behavioral alignment and robustness to perturbations in other domains.

Abstract

As AI systems are deployed across increasingly diverse and high-stakes settings, model alignment must generalize beyond the tasks and domains seen during training. This is especially important for reinforcement learning (RL), which can introduce unexpected misalignment through reward hacking, deception, or other unintended strategies. We study whether RL on beneficial behavior, instantiated in realistic domains, can produce broad and persistent alignment generalization beyond the training distribution. We construct a dataset of realistic situations designed to measure and train beneficial traits, such as truthfulness, fairness, risk awareness, and corrigibility, spanning varied domains, including health, science, and education. We then train models with RL on this dataset and evaluate them on more than 50 independent benchmarks of alignment and beneficial behavior. Compared to a compute-matched baseline, beneficial trait RL improves performance on over 80% of these out-of-distribution benchmarks. We observe substantial out-of-distribution alignment transfer: a beneficial-behavior RL intervention entirely limited to one domain, health, produces broad improvements on non-health alignment evaluations, including reduced reward hacking, deception, and general misalignment. Finally, we study alignment persistence: whether behavior remains robustly aligned under attempts to steer models towards misalignment. Models trained with beneficial trait RL show improved persistence, including greater resistance to adversarial prompting and harmful finetuning; further work is required to isolate the sources of these effects. These results suggest that RL to reinforce beneficial behavior in realistic domains can produce models that are more robustly aligned with human flourishing.

Problem

• reinforcement learning
• alignment generalization
• beneficial behavior
• out-of-distribution
• alignment persistence

Innovation

• reinforcement learning
• alignment generalization
• beneficial behavior
• out-of-distribution robustness
• alignment persistence