A Reward-Petri-Net Interpretation of Temporal Behavior Trees

2026-06-19

AI Summary

This work addresses the challenge of reward function design in complex, long-horizon robotic tasks by proposing an automatic reward generation method based on Temporal Behavior Trees (TBTs). It establishes, for the first time, a formal mapping between TBTs and Reward Petri Nets (RPNs), incorporates Linear Temporal Logic at leaf nodes to encode temporal constraints, and leverages the hierarchical structure of TBTs to automatically generate structure-aware reinforcement learning reward signals. The approach significantly improves sample efficiency and task scalability, outperforming conventional reinforcement learning baselines across multiple challenging environments. Furthermore, it enables flexible control over the learning process through diverse reward allocation strategies.

Abstract

This paper introduces an interpretation of Temporal Behavior Trees (TBTs) as Reward-Petri-Nets (RPNs) for reinforcement learning (RL). Designing reward functions for complex, long-horizon robotic tasks is notoriously difficult, especially when tasks have hierarchical structure and temporal constraints. TBTs extend conventional behavior trees (BTs) used in robotic applications by incorporating temporal properties into their leaf nodes. This allows TBTs to represents not only the behavioral task structure defined by BT operators such as Sequence, Fallback, and Parallel, but also the task's temporal constraints. In this work, the constraints are specified in the leaf nodes using Linear Temporal Logic. In order to inform RL rewards using TBTs, we provide a translation from TBT into a Petri Net (PN) and show how rewards can be automatically assigned based on the TBT's structure, resulting in a RPN. In a series of increasingly challenging environments, we demonstrate how TBT-based rewards enable learning where vanilla RL fails, improve sample efficiency, and offer flexible, intuitive control over the learning progress. We showcase the learning impact by using different reward distribution schemes and TBT structures.

Problem

• reward design
• temporal constraints
• hierarchical tasks
• reinforcement learning
• robotic tasks

Innovation

• Temporal Behavior Trees
• Reward-Petri-Nets
• Reinforcement Learning
• Linear Temporal Logic
• Hierarchical Task Structure