🤖 AI Summary
This work addresses the challenges of modularity, interpretability, and safety in reinforcement learning (RL) task specification, with particular emphasis on compositional robustness under functional decomposition. We propose the first category-theoretic framework for RL compositionality: defining the category of Markov decision processes (MDPs), and introducing categorical constructions—including fiber products, coproducts, and pushouts—to formally characterize subtask decomposition, policy composition, and unsafe state elimination. We further pioneer the use of categorical semantics to unify modeling of state-action symmetry embeddings, sequential task concatenation, and chained task completion—represented via zig-zag diagrams. The framework rigorously establishes sufficient conditions under which “divide-and-conquer” learning yields globally optimal policies. By grounding modular RL in rigorous algebraic semantics, it enables verifiable, composable RL system design.
📝 Abstract
Reinforcement learning (RL) often requires decomposing a problem into subtasks and composing learned behaviors on these tasks. Compositionality in RL has the potential to create modular subtask units that interface with other system capabilities. However, generating compositional models requires the characterization of minimal assumptions for the robustness of the compositional feature. We develop a framework for a emph{compositional theory} of RL using a categorical point of view. Given the categorical representation of compositionality, we investigate sufficient conditions under which learning-by-parts results in the same optimal policy as learning on the whole. In particular, our approach introduces a category $mathsf{MDP}$, whose objects are Markov decision processes (MDPs) acting as models of tasks. We show that $mathsf{MDP}$ admits natural compositional operations, such as certain fiber products and pushouts. These operations make explicit compositional phenomena in RL and unify existing constructions, such as puncturing hazardous states in composite MDPs and incorporating state-action symmetry. We also model sequential task completion by introducing the language of zig-zag diagrams that is an immediate application of the pushout operation in $mathsf{MDP}$.
