Stratifying Reinforcement Learning with Signal Temporal Logic

2026-04-06

AI Summary

This work investigates the geometric and structural organization of embedding spaces in deep reinforcement learning, establishing a formal connection to temporal task specifications. By introducing a hierarchical framework, it proposes a novel semantics for Signal Temporal Logic (STL), interpreting atomic predicates as membership tests within layered subspaces and revealing how STL formulas induce hierarchical spatiotemporal structure. This theoretical foundation not only enables rigorous analysis of embedding space geometry but also inspires the design of efficient computational methods. Experiments in Minigrid environments empirically validate the hierarchical nature of learned embeddings, and the proposed signature-based approach effectively captures their intrinsic structure, demonstrating initial promise for enhancing interpretability and performance in reinforcement learning representations.

Abstract

In this paper, we develop a stratification-based semantics for Signal Temporal Logic (STL) in which each atomic predicate is interpreted as a membership test in a stratified space. This perspective reveals a novel correspondence principle between stratification theory and STL, showing that most STL formulas can be viewed as inducing a stratification of space-time. The significance of this interpretation is twofold. First, it offers a fresh theoretical framework for analyzing the structure of the embedding space generated by deep reinforcement learning (DRL) and relates it to the geometry of the ambient decision space. Second, it provides a principled framework that both enables the reuse of existing high-dimensional analysis tools and motivates the creation of novel computational techniques. To ground the theory, we (1) illustrate the role of stratification theory in Minigrid games and (2) apply numerical techniques to the latent embeddings of a DRL agent playing such a game where the robustness of STL formulas is used as the reward. In the process, we propose computationally efficient signatures that, based on preliminary evidence, appear promising for uncovering the stratification structure of such embedding spaces.

Problem

• Reinforcement Learning
• Signal Temporal Logic
• Stratification
• Embedding Space
• Decision Space

Innovation

• Stratification
• Signal Temporal Logic
• Deep Reinforcement Learning
• Embedding Space
• Robustness