Objective-Behavior Alignment: Diagnostics for MORL Policy Selection

2026-06-19

AI Summary

This work addresses a critical limitation in multi-objective reinforcement learning, where policy evaluation based solely on value vectors often overlooks behavioral differences, leading to ambiguous decision-making. To resolve this, the authors propose an exploratory diagnostic framework that explicitly incorporates behavioral divergence into Pareto front analysis for the first time. By integrating trajectory clustering with visualization techniques, the method quantitatively reveals the behavioral diversity among Pareto-optimal policies. Empirical validation on both grid-world and continuous control benchmarks demonstrates its effectiveness: even in complex tasks, the framework clearly delineates behavioral distinctions between policies, thereby offering decision-makers a richer, more informative basis for policy selection.

Abstract

Real-world decision-making often requires optimizing multiple competing objectives simultaneously. In reinforcement learning (RL), this is typically addressed by combining reward signals into a single scalar objective via a scalarization function, which can be fragile: small changes in the weights can induce drastically different policies. Multi-objective reinforcement learning (MORL) instead produces sets of policies that explicitly represent trade-offs between objectives. However, these policies are typically presented to the decision maker only through their value vectors, which can obscure substantial behavioral variation: policies that induce distinct trajectories may appear indistinguishable when evaluated solely by expected returns. We propose an exploratory diagnostic workflow that automatically highlights behavioral variation along the Pareto front that objective values alone do not reveal, providing both quantitative and visual tools to support policy inspection. We validate our approach on simple grid examples and scale it to continuous control benchmarks, demonstrating that it remains effective as problem complexity increases.

Problem

• multi-objective reinforcement learning
• policy selection
• behavioral variation
• Pareto front
• objective-behavior alignment

Innovation

• Multi-objective Reinforcement Learning
• Pareto Front
• Behavioral Diversity
• Policy Diagnostics
• Scalarization