AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL

2026-06-30

AI Summary

In nonlinear multi-objective reinforcement learning, the scalarized expected return (SER) and expected scalarized return (ESR) paradigms have long remained disjoint due to fundamental differences in their optimization levels and policy characteristics, hindering unified modeling of complex preferences such as risk aversion or fairness. This work proposes the Aggregation-Expectation-Transformation (AET) framework, which reconciles SER and ESR through a tripartite decomposition and introduces AETDICE—the first nonlinear multi-objective offline reinforcement learning algorithm capable of training from static datasets. By integrating DICE-style density ratio estimation within an augmented state space, AETDICE enables effective off-policy optimization and overcomes the longstanding trade-off limitations between SER and ESR in existing approaches.

Abstract

Optimizing nonlinear preferences in multi-objective reinforcement learning (MORL) is essential for capturing complex trade-offs like risk aversion or fairness. However, such non-linearity has historically bifurcated nonlinear MORL objectives into two distinct paradigms: Scalarized Expected Return (SER) and Expected Scalarized Return (ESR). While SER requires global-level optimization and ESR requires non-Markovian policies, leading to fragmented optimization strategies, we bridge this divide through the Aggregation-Expectation-Transformation (AET) framework. By unifying both criteria through a tripartite decomposition of scalarization, AET provides a principled foundation for general nonlinear MORL. Building on this framework, we propose AETDICE, a tractable offline RL algorithm for AET objectives. By utilizing DICE-style density-ratio estimation in an augmented state space, AETDICE enables sample-based optimization from static datasets. Our framework resolves long-standing barriers and captures respective trade-offs induced by AET framework, which existing methods fail to address.

Problem

• multi-objective reinforcement learning
• nonlinear preferences
• Scalarized Expected Return
• Expected Scalarized Return
• offline optimization

Innovation

• Nonlinear Multi-Objective RL
• AET Framework
• Offline Reinforcement Learning
• DICE Estimation
• Scalarization