Performance-Driven Environment Abstraction with Multi-Timescale Learning

2026-06-15

AI Summary

This work addresses the limitations of existing environment abstraction methods for large-scale Markov decision processes, which typically prioritize geometric or topological fidelity while overlooking their direct impact on policy performance. The authors propose a policy-performance-oriented tree-structured abstraction mechanism that constructs controllable approximations through state clustering and intra-cluster action distribution sharing. The abstraction granularity is dynamically adjusted based on Q-value discrepancies. Furthermore, the approach explicitly decouples value function approximation error from action-sharing loss and integrates multi-timescale reinforcement learning to enable adaptive refinement and coarsening of the abstract structure. This method achieves substantial state-space compression and demonstrates superior sample efficiency and replanning speed compared to standard actor-critic baselines.

Abstract

We study performance-driven environment abstraction for decision-making in large Markov decision processes. Rather than preserving geometric or topological structure, we seek abstractions that directly optimize decision quality. We model abstraction as a controlled approximation obtained by aggregating the state space and enforcing a shared action distribution within each aggregated state. For a fixed partition, we establish a performance guarantee that separates value-function approximation error from the loss introduced by action sharing. Guided by this analysis, we develop a multi-timescale reinforcement learning framework that jointly adapts the policy and a tree-structured environment abstraction. The resulting algorithm refines and coarsens regions of the state space based on Q-value discrepancies, balancing performance against abstraction size and complexity. Empirical results demonstrate substantial state compression, improved sample efficiency, and faster replanning compared to actor-critic baselines.

Problem

• environment abstraction
• Markov decision processes
• performance-driven
• state aggregation
• decision-making

Innovation

• performance-driven abstraction
• multi-timescale reinforcement learning
• state aggregation
• action sharing
• tree-structured abstraction