Limitations of Scalarisation in MORL: A Comparative Study in Discrete Environments

2025-11-20

AI Summary

This study systematically exposes fundamental limitations of scalarization-based methods in multi-objective reinforcement learning (MORL) with discrete action and observation spaces: poor Pareto-front coverage, low robustness, and strong dependence on environmental properties and front geometry. To address these issues, we propose an inner-loop multi-policy architecture and comparatively evaluate three representative approaches—linear scalarization, Chebyshev scalarization, and non-scalarized Pareto Q-learning—under both outer-loop single-policy and inner-loop multi-policy paradigms. Results demonstrate that Pareto Q-learning significantly improves solution-set diversity and stability over scalarized methods. Moreover, the inner-loop multi-policy design effectively mitigates scalarization’s sensitivity to weight selection and susceptibility to local optima. Our empirical analysis provides both a novel methodological paradigm and rigorous evidence for developing robust, scalable MORL algorithms.

Technology Category

• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Adversarial Learning
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Scalarisation functions are widely employed in MORL algorithms to enable intelligent decision-making. However, these functions often struggle to approximate the Pareto front accurately, rendering them unideal in complex, uncertain environments. This study examines selected Multi-Objective Reinforcement Learning (MORL) algorithms across MORL environments with discrete action and observation spaces. We aim to investigate further the limitations associated with scalarisation approaches for decision-making in multi-objective settings. Specifically, we use an outer-loop multi-policy methodology to assess the performance of a seminal single-policy MORL algorithm, MO Q-Learning implemented with linear scalarisation and Chebyshev scalarisation functions. In addition, we explore a pioneering inner-loop multi-policy algorithm, Pareto Q-Learning, which offers a more robust alternative. Our findings reveal that the performance of the scalarisation functions is highly dependent on the environment and the shape of the Pareto front. These functions often fail to retain the solutions uncovered during learning and favour finding solutions in certain regions of the solution space. Moreover, finding the appropriate weight configurations to sample the entire Pareto front is complex, limiting their applicability in uncertain settings. In contrast, inner-loop multi-policy algorithms may provide a more sustainable and generalizable approach and potentially facilitate intelligent decision-making in dynamic and uncertain environments.

Problem

• Scalarisation functions poorly approximate Pareto fronts in complex environments
• Finding proper weight configurations for full Pareto sampling is challenging
• Multi-policy algorithms offer more robust alternatives to scalarisation approaches

Innovation

• Compare scalarisation functions in discrete MORL environments
• Evaluate outer-loop multi-policy MO Q-Learning algorithms
• Explore inner-loop Pareto Q-Learning for robust solutions