Homomorphic Mappings for Value-Preserving State Aggregation in Markov Decision Processes

2025-10-10

AI Summary

This paper addresses the policy performance degradation caused by state aggregation in Markov decision processes (MDPs). To preserve optimality under abstraction, we propose a homomorphism-based optimality-preserving abstraction framework. Its core contribution is establishing sufficient conditions for *optimal policy equivalence*, guaranteeing that policies optimized in the low-dimensional abstract MDP remain optimal—or near-optimal within a controllable error bound—in the original MDP. We design Homomorphic Policy Gradient (HPG) and its enhanced variant EBHPG, providing theoretical guarantees on convergence, approximation error bounds, and policy performance lower bounds. The method integrates homomorphic abstraction theory, linear value function approximation, and policy gradient optimization, supported by rigorous error analysis to ensure generalization. Empirical evaluation across multi-task domains demonstrates that our approach achieves a superior trade-off between computational efficiency and policy performance, outperforming seven baseline algorithms.

Technology Category

• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization
• Multiagent Systems: Multiagent Learning

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

State aggregation aims to reduce the computational complexity of solving Markov Decision Processes (MDPs) while preserving the performance of the original system. A fundamental challenge lies in optimizing policies within the aggregated, or abstract, space such that the performance remains optimal in the ground MDP-a property referred to as {"}optimal policy equivalence {"}. This paper presents an abstraction framework based on the notion of homomorphism, in which two Markov chains are deemed homomorphic if their value functions exhibit a linear relationship. Within this theoretical framework, we establish a sufficient condition for the equivalence of optimal policy. We further examine scenarios where the sufficient condition is not met and derive an upper bound on the approximation error and a performance lower bound for the objective function under the ground MDP. We propose Homomorphic Policy Gradient (HPG), which guarantees optimal policy equivalence under sufficient conditions, and its extension, Error-Bounded HPG (EBHPG), which balances computational efficiency and the performance loss induced by aggregation. In the experiments, we validated the theoretical results and conducted comparative evaluations against seven algorithms.

Problem

• Reducing MDP computational complexity while preserving performance
• Ensuring optimal policy equivalence between abstract and ground MDPs
• Balancing computational efficiency with performance loss in aggregation

Innovation

• Homomorphic mappings enable value-preserving state aggregation
• HPG guarantees optimal policy equivalence under sufficient conditions
• EBHPG balances computational efficiency with performance loss