Multi-Objective Planning with Contextual Lexicographic Reward Preferences

2025-02-13

AI Summary

Existing multi-objective planning methods struggle to enable agents to dynamically switch among multiple lexicographic preference orderings in response to state changes in dynamic environments. Method: We propose the Context-aware Lexicographic Markov Decision Process (CLMDP) framework, the first to jointly integrate context inference, state-to-context mapping modeling, and multi-objective policy synthesis within a lexicographic optimization paradigm—enabling acyclic, unified policy learning across contexts. Our approach combines Bayesian context inference with multi-objective policy synthesis to adaptively generate piecewise-optimal policies satisfying distinct lexicographic preference orderings. Results: Evaluated on both simulation benchmarks and a real mobile robot platform, CLMDP significantly improves safety, execution efficiency, and preference consistency under multi-context tasks. It establishes a scalable theoretical and practical foundation for dynamic multi-objective autonomous decision-making.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Multiagent Systems: Multiagent Planning
• Machine Learning: Learning Preferences or Rankings

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Autonomous agents are often required to plan under multiple objectives whose preference ordering varies based on context. The agent may encounter multiple contexts during its course of operation, each imposing a distinct lexicographic ordering over the objectives, with potentially different reward functions associated with each context. Existing approaches to multi-objective planning typically consider a single preference ordering over the objectives, across the state space, and do not support planning under multiple objective orderings within an environment. We present Contextual Lexicographic Markov Decision Process (CLMDP), a framework that enables planning under varying lexicographic objective orderings, depending on the context. In a CLMDP, both the objective ordering at a state and the associated reward functions are determined by the context. We employ a Bayesian approach to infer a state-context mapping from expert trajectories. Our algorithm to solve a CLMDP first computes a policy for each objective ordering and then combines them into a single context-aware policy that is valid and cycle-free. The effectiveness of the proposed approach is evaluated in simulation and using a mobile robot.

Problem

• Multi-objective planning with varying preferences
• Contextual lexicographic reward functions
• State-context mapping from expert trajectories

Innovation

• Contextual Lexicographic Markov Decision Process
• Bayesian state-context mapping inference
• Context-aware policy combination