🤖 AI Summary
This work addresses the high sample cost that limits reinforcement learning in robotic applications by proposing a novel dual-network pretraining framework for Actor-Critic algorithms such as PPO. Unlike existing approaches that pretrain only the policy network (Actor), this method simultaneously initializes both the Actor and the value network (Critic) using expert demonstration data, thereby significantly improving sample efficiency. By integrating behavioral cloning with reward estimation, the approach achieves a 86.1% improvement in sample efficiency over no pretraining and a 30.9% gain compared to Actor-only pretraining across 15 simulated robotic tasks. These results demonstrate the effectiveness and novelty of jointly pretraining both networks in an Actor-Critic architecture.
📝 Abstract
Reinforcement learning (RL) actor-critic algorithms enable autonomous learning but often require a large number of environment interactions, which limits their applicability in robotics. Leveraging expert data can reduce the number of required environment interactions. A common approach is actor pretraining, where the actor network is initialized via behavioral cloning on expert demonstrations and subsequently fine-tuned with RL. In contrast, the initialization of the critic network has received little attention, despite its central role in policy optimization. This paper proposes a pretraining approach for actor-critic algorithms like Proximal Policy Optimization (PPO) that uses expert demonstrations to initialize both networks. The actor is pretrained via behavioral cloning, while the critic is pretrained using returns obtained from rollouts of the pretrained policy. The approach is evaluated on 15 simulated robotic manipulation and locomotion tasks. Experimental results show that actor-critic pretraining improves sample efficiency by 86.1% on average compared to no pretraining and by 30.9% to actor-only pretraining.