Belief-State Query Policies for User-Aligned POMDPs

2024-05-24

Neural Information Processing Systems

AI Summary

Planning under partial observability while ensuring alignment with user preferences remains challenging due to the difficulty of formally encoding and optimizing user constraints within POMDPs. Method: This paper proposes a Parameterized Belief State Query (BSQ) policy, formalizing user constraints and preferences within a goal-directed POMDP framework. We provide the first theoretical analysis of BSQ, proving that its expected cost function is piecewise-constant—thereby yielding a finite, discrete parameter space—and design a guaranteed-convergent algorithm to optimal user-aligned behavior. The approach integrates generalized POMDP (gPOMDP) modeling, parametric policy optimization, piecewise-constant function analysis, and implicit discrete search. Results: Experiments demonstrate computationally feasible partially observable planning strictly satisfying user alignment; theoretical analysis proves convergence to optimal user-aligned policies. The core innovation lies in embedding user alignment directly into the POMDP policy structure and establishing an analytically tractable, optimizable parametric belief querying mechanism.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Planning in real-world settings often entails addressing partial observability while aligning with users' requirements. We present a novel framework for expressing users' constraints and preferences about agent behavior in a partially observable setting using parameterized belief-state query (BSQ) policies in the setting of goal-oriented partially observable Markov decision processes (gPOMDPs). We present the first formal analysis of such constraints and prove that while the expected cost function of a parameterized BSQ policy w.r.t its parameters is not convex, it is piecewise constant and yields an implicit discrete parameter search space that is finite for finite horizons. This theoretical result leads to novel algorithms that optimize gPOMDP agent behavior with guaranteed user alignment. Analysis proves that our algorithms converge to the optimal user-aligned behavior in the limit. Empirical results show that parameterized BSQ policies provide a computationally feasible approach for user-aligned planning in partially observable settings.

Problem

• Addressing partial observability in real-world planning
• Aligning agent behavior with user constraints and preferences
• Optimizing gPOMDPs for guaranteed user-aligned outcomes

Innovation

• Parameterized belief-state query policies for gPOMDPs
• Non-convex but piecewise constant cost function
• Finite discrete parameter search space algorithms