Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation

2026-06-16

AI Summary

This work addresses the critical yet underexplored impact of action space selection on motion smoothness, safety, and task performance in vision-based robotic manipulation. For the first time, it systematically benchmarks four action spaces—pose deltas, pose velocities, joint position deltas, and joint velocities—within a vision-driven reinforcement learning framework. The evaluation spans both simulation training and sim-to-real transfer, with comprehensive experiments on object grasping and pushing tasks. Results demonstrate that the joint velocity action space significantly enhances task success rates and motion smoothness on real robots, offering crucial empirical guidance for action space design in practical vision-based manipulation systems.

Abstract

In real-world reinforcement learning (RL), the choice of action space can play a key role in shaping motion smoothness, safety, and overall task performance. In this study, we evaluate pose increment, pose velocity, joint position increment, and joint velocity across two vision-based manipulation tasks: object picking and pushing. We train policies in simulation and deploy them to the real world using sim-to-real transfer. We find that action-space representation indeed significantly affects sim-to-real performance. In particular, we find that the joint velocity action space is best for the vision-based picking and pushing tasks in terms of smoothness and final task performance. We also provide practical guidance for RL practitioners in choosing action spaces for both simulation and real-world experiments.

Problem

• action space
• reinforcement learning
• vision-based manipulation
• sim-to-real transfer
• robotic control

Innovation

• action space
• sim-to-real transfer
• vision-based manipulation
• reinforcement learning
• joint velocity