Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning

2026-06-24

AI Summary

This work addresses the challenge in multi-objective reinforcement learning that traditional scalarization methods often fail to fully capture the Pareto optimal front. To overcome this limitation, the paper proposes a preference-conditioned Bellman operator based on Chebyshev scalarization, embedded within an end-to-end framework for learning deterministic Pareto optimal policies. The proposed operator possesses an envelope property that guarantees the value function’s upper bound encompasses the true Pareto front and ensures monotonic convergence. Consequently, it enables the synthesis of approximately Pareto optimal policies under arbitrary user-specified preferences. Experimental results demonstrate that the method effectively recovers complex trade-offs among multiple objectives and achieves comprehensive coverage of the Pareto front.

Abstract

Real-world decision-making often requires balancing multiple conflicting objectives, a challenge that standard Reinforcement Learning (RL) frequently addresses by aggregating rewards into a single scalar signal. While effective for simple tasks, this approach often fails to capture the full spectrum of optimal trade-offs, known as the Pareto frontier. In this paper, we introduce a novel preference-conditioned Bellman operator, motivated from the Chebyshev scalarization, designed to compute deterministic Pareto-optimal policies for Multi-Objective Markov Decision Processes (MOMDPs). We prove that this operator satisfies an enveloping property, where the estimated value functions upper-bound the true Pareto frontier, and demonstrate that it monotonically converges to a coverage set of this frontier. Furthermore, we also show how to extract deterministic policies from these converged Q-estimates. This ensures the agent can recover a policy for any given preference, capturing the entire Pareto-optimal frontier while guaranteeing each synthesized policy remains approximately Pareto-optimal. Experimental results validate that our algorithm successfully recovers complex trade-offs, providing a solution for deterministic Pareto-optimal policy synthesis.

Problem

• Multi-Objective Reinforcement Learning
• Pareto-Optimal Policy
• Deterministic Policy
• Pareto Frontier
• MOMDP

Innovation

• Pareto-optimal policy
• multi-objective reinforcement learning
• preference-conditioned Bellman operator
• Chebyshev scalarization
• deterministic policy synthesis