Multi-objective Reinforcement Learning With Augmented States Requires Rewards After Deployment

2026-04-17

AI Summary

This study addresses a critical yet overlooked limitation in multi-objective reinforcement learning (MORL): existing methods that handle nonlinear utility functions by augmenting the state with historical cumulative rewards implicitly depend on continuous access to reward signals during deployment. The work explicitly identifies and systematically analyzes this post-deployment reward dependency, revealing its practical constraints in real-world applications. By integrating insights from MORL, augmented state modeling, and nonlinear utility theory, the paper uncovers hidden assumptions underlying current algorithms and demonstrates how they hinder deployment in settings where reward information is unavailable after training. These findings provide essential theoretical grounding and practical guidance for developing truly deployable MORL approaches that operate without requiring reward signals at test time.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Multiagent Learning
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This research note identifies a previously overlooked distinction between multi-objective reinforcement learning (MORL), and more conventional single-objective reinforcement learning (RL). It has previously been noted that the optimal policy for an MORL agent with a non-linear utility function is required to be conditioned on both the current environmental state and on some measure of the previously accrued reward. This is generally implemented by concatenating the observed state of the environment with the discounted sum of previous rewards to create an augmented state. While augmented states have been widely-used in the MORL literature, one implication of their use has not previously been reported -- namely that they require the agent to have continued access to the reward signal (or a proxy thereof) after deployment, even if no further learning is required. This note explains why this is the case, and considers the practical repercussions of this requirement.

Problem

• multi-objective reinforcement learning
• augmented states
• reward signal
• deployment
• policy conditioning

Innovation

• multi-objective reinforcement learning
• augmented states
• reward signal after deployment
• non-linear utility function
• policy conditioning