LTL-Constrained Policy Optimization with Cycle Experience Replay

2024-04-17

AI Summary

In continuous state-action spaces, sparse Linear Temporal Logic (LTL) constraints hinder policy optimization, as agents struggle to simultaneously satisfy LTL specifications and maximize reward. Method: We propose Cycle Experience Replay (CyclER), the first experience replay mechanism that explicitly incorporates the structure of the LTL automaton. CyclER identifies constraint-relevant sub-behaviors via cyclic trajectory sampling over automaton states and applies progressive reward shaping guided by automaton progression. The method integrates LTL automaton construction, function-approximation-based reinforcement learning, and structured reward reshaping tailored for sparse LTL constraints. Results: Evaluated on three continuous-control benchmarks, CyclER substantially outperforms existing reward-shaping approaches—achieving higher cumulative rewards while strictly satisfying all LTL constraints—thereby effectively mitigating training instability and sample inefficiency induced by constraint sparsity.

Technology Category

• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Machine Learning: Reinforcement Learning
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Linear Temporal Logic (LTL) offers a precise means for constraining the behavior of reinforcement learning agents. However, in many tasks, LTL is insufficient for task specification; LTL-constrained policy optimization, where the goal is to optimize a scalar reward under LTL constraints, is needed. Prior methods for this constrained problem are restricted to finite state spaces. In this work, we present Cycle Experience Replay (CyclER), a reward-shaping approach to this problem that allows continuous state and action spaces and the use of function approximations. CyclER guides a policy towards satisfaction by encouraging partial behaviors compliant with the LTL constraint, using the structure of the constraint. In doing so, it addresses the optimization challenges stemming from the sparse nature of LTL satisfaction. We evaluate CyclER in three continuous control domains. On these tasks, CyclER outperforms existing reward-shaping methods at finding performant and LTL-satisfying policies.

Problem

• Optimizing scalar rewards under LTL constraints in reinforcement learning
• Addressing sparse reward issues in LTL-constrained policy optimization
• Encouraging constraint-compliant behaviors through structured reward shaping

Innovation

• Cycle Experience Replay for LTL constraints
• Reward shaping using partial constraint compliance
• Theoretical guarantee for near-optimal satisfaction probability