Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement

2026-06-17

AI Summary

Existing Vision-Language-Action (VLA) models struggle to achieve zero-shot sim-to-real transfer in physical interaction tasks due to error accumulation during execution. This work proposes an object-centric residual reinforcement learning framework that builds upon a frozen base VLA policy by constructing a compact, simulation-to-reality consistent observation space using object poses. Domain discrepancies are mitigated through replaying teleoperated data in simulation, aligning the visual and dynamics gaps without real-world fine-tuning. The approach innovatively integrates object poses into residual RL and combines pose noise with dropout training to enable robust zero-shot transfer. Evaluated on five manipulation tasks with a Franka FR3 robot, the method improves zero-shot success rates from 42% to 76%. Furthermore, it enables self-iterative refinement of the base VLA model using improved trajectories, without requiring additional teleoperated demonstrations.

Abstract

Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/

Problem

• Vision-Language-Action
• Sim-to-Real
• Residual RL
• Zero-Shot Transfer
• Object-Centric

Innovation

• Object-Centric RL
• Residual Reinforcement Learning
• Zero-Shot Sim-to-Real Transfer
• Vision-Language-Action Models
• Pose-Based Observation