Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning

2026-06-16

AI Summary

This work addresses the challenge in multi-objective reinforcement learning (MORL) of simultaneously achieving Pareto optimality and fairness under unknown or dynamically changing user preferences, while also lacking a diverse set of fair policies. To this end, the authors propose a unified multi-policy learning framework that integrates generalized Gini welfare functions with non-stationary and stochastic policies. They theoretically show that under concave piecewise-linear welfare functions, the set of fair policies forms a convex coverage set. Building on this insight, they develop three novel algorithms: state-augmented non-stationary policy learning, stochastic policy optimization, and convex coverage set approximation. Experimental results across multiple benchmark environments demonstrate that the proposed approach significantly outperforms existing MORL baselines, successfully generating a diverse set of fair, Pareto-optimal policies that span a broad spectrum of user preferences.

Abstract

Fairness is an important aspect of decision-making in multi-objective reinforcement learning (MORL), where policies must ensure both optimality and equity across multiple, potentially conflicting objectives. While single-policy MORL methods can learn fair policies for fixed user preferences using welfare functions such as the generalized Gini welfare function (GGF), they fail to provide the diverse set of policies necessary for dynamic or unknown user preferences. To address this limitation, we formalize the fair optimization problem in multi-policy MORL, where the goal is to learn a set of Pareto-optimal policies that ensure fairness across all possible user preferences. Our key technical contributions are threefold: (1) We show that for concave, piecewise-linear welfare functions (e.g., GGF), fair policies remain in the convex coverage set (CCS), which is an approximated Pareto front for linear scalarization. (2) We demonstrate that non-stationary policies, augmented with accrued reward histories, and stochastic policies improve fairness by dynamically adapting to historical inequities. (3) We propose three novel algorithms, which include integrating GGF with multi-policy multi-objective Q-Learning (MOQL), state-augmented multi-policy MOQL for learning non-statoinary policies, and its novel extension for learning stochastic policies. We evaluate our algorithms across various domains and compare our methods against the state-of-the-art MORL baselines. The empirical results show that our methods learn a set of fair policies that accommodate different user preferences.

Problem

• Fairness
• Pareto-Optimal Policies
• Multi-Objective Reinforcement Learning
• User Preferences
• Welfare Functions

Innovation

• Fairness
• Pareto-Optimal Policies
• Multi-Objective Reinforcement Learning
• Generalized Gini Welfare Function
• Non-stationary Policies