Reward Machines for Signal Temporal Logic

2026-08-13

AI Summary

This study addresses the challenges of state explosion due to history dependence and long-horizon control in Signal Temporal Logic (STL) reinforcement learning. We propose a memory-augmented mechanism based on timed alternating automata for model-free systems. By formulating the problem as a Markov Reward Process, this approach effectively overcomes state expansion bottlenecks associated with nested temporal operators, enabling efficient policy learning. Experimental results demonstrate that the proposed method significantly outperforms existing robustness reward-based approaches in both policy robustness scores and task satisfaction rates. Consequently, this work provides a superior solution for reinforcement learning under complex temporal logic constraints, offering enhanced performance where traditional methods struggle with historical dependencies and extended control horizons.

Abstract

Signal temporal logic (STL) provides a formal language for specifying real-time properties of real-valued observations, along with a quantitative robustness score for monitoring satisfaction. Control synthesis from STL specifications is of interest since manual controller design becomes infeasible as real-world systems grow in complexity. Moreover, many modern autonomous and AI-enabled systems lack accurate and complete system models, which makes optimization-based synthesis approaches unsuitable and motivates learning-based control. Prior work uses STL robustness scores as rewards in reinforcement learning (RL) to obtain control policies satisfying given specifications; however, robustness depends on execution history, leading to intractable state space expansion for general long-horizon specifications with arbitrarily nested temporal operators. This work introduces a novel automata-based approach that provides an efficient memory mechanism and associated Markovian rewards suitable for RL frameworks. Our approach constructs a timed alternating automaton from the given STL specifications, augments the state space with automaton locations and clock valuations, and derives rewards from the automaton acceptance condition. We empirically demonstrate that our approach learns policies that achieve higher robustness scores and satisfaction rates than those learned by existing approaches using robustness-based rewards.

Problem

• Signal Temporal Logic
• Reinforcement Learning
• State Space Explosion
• Non-Markovian Rewards
• Control Synthesis

Innovation

• Signal Temporal Logic
• Reward Machines
• Timed Alternating Automata
• Reinforcement Learning
• Markovian Rewards