🤖 AI Summary
To address the challenges of sparse rewards and poor scalability to high-dimensional continuous state spaces in reinforcement learning (RL) under Linear Temporal Logic (LTL) specifications, this paper proposes an intrinsic reward shaping method grounded in Limit-Deterministic Büchi Automata (LDBAs). First, the LDBA is formalized as a Markov reward process, and a Bayesian dynamics prior is introduced to enable interpretable, high-level state-value estimation. Second, this estimated value function is transformed into a potential function to construct shaped intrinsic rewards that guide efficient exploration. The approach significantly improves LTL task completion rates across both discrete and high-dimensional continuous control benchmarks. It successfully solves long-horizon, multi-objective LTL planning problems and achieves 3–5× higher sample efficiency compared to prior LTL-RL frameworks, thereby overcoming key scalability limitations in existing LTL-guided RL methods.
📝 Abstract
Linear temporal logic (LTL) is a powerful language for task specification in reinforcement learning, as it allows describing objectives beyond the expressivity of conventional discounted return formulations. Nonetheless, recent works have shown that LTL formulas can be translated into a variable rewarding and discounting scheme, whose optimization produces a policy maximizing a lower bound on the probability of formula satisfaction. However, the synthesized reward signal remains fundamentally sparse, making exploration challenging. We aim to overcome this limitation, which can prevent current algorithms from scaling beyond low-dimensional, short-horizon problems. We show how better exploration can be achieved by further leveraging the LTL specification and casting its corresponding Limit Deterministic B""uchi Automaton (LDBA) as a Markov reward process, thus enabling a form of high-level value estimation. By taking a Bayesian perspective over LDBA dynamics and proposing a suitable prior distribution, we show that the values estimated through this procedure can be treated as a shaping potential and mapped to informative intrinsic rewards. Empirically, we demonstrate applications of our method from tabular settings to high-dimensional continuous systems, which have so far represented a significant challenge for LTL-based reinforcement learning algorithms.