Sample-efficient Neuro-symbolic Proximal Policy Optimization

2026-04-28

AI Summary

This work addresses the challenges of low sample efficiency and learning difficulty in deep reinforcement learning when dealing with sparse rewards, long horizons, and multi-subtask structures. The authors propose integrating symbolic logic policy specifications into Proximal Policy Optimization (PPO) through two neuro-symbolic guidance mechanisms: H-PPO-Product biases the action distribution to guide exploration, while H-PPO-SymLoss incorporates a symbolic regularization term into the loss function, enabling effective learning even with imperfect prior knowledge. Leveraging logical specifications extracted by reward machines, the approach facilitates policy transfer and regularization. Empirical results on OfficeWorld, WaterWorld, and DoorKey benchmarks demonstrate that both variants significantly outperform standard PPO and reward machine baselines, achieving faster convergence and higher final returns.

Abstract

Deep Reinforcement Learning (DRL) algorithms often require a large amount of data and struggle in sparse-reward domains with long planning horizons and multiple sub-goals. In this paper, we propose a neuro-symbolic extension of Proximal Policy Optimization (PPO) that transfers partial logical policy specifications learned in easier instances to guide learning in more challenging settings. We introduce two integrations of symbolic guidance: (i) H-PPO-Product, which biases the action distribution at sampling time, and (ii) H-PPO-SymLoss, which augments the PPO loss with a symbolic regularization term. We evaluate our methods on three benchmarks (OfficeWorld, WaterWorld, and DoorKey), showing consistently faster learning and higher return at convergence than PPO and a Reward Machine baseline, also under imperfect symbolic knowledge.

Problem

• sample efficiency
• sparse-reward domains
• long planning horizons
• multiple sub-goals
• deep reinforcement learning

Innovation

• neuro-symbolic reinforcement learning
• Proximal Policy Optimization
• symbolic guidance
• sample efficiency
• logical policy specification