Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning

2026-05-19

AI Summary

This work addresses the challenge of online adaptation for pretrained imitation policies in real-world robotic deployment, where execution errors and distribution shifts necessitate efficient fine-tuning. Existing action-space residual methods suffer from high exploration noise and low sample efficiency. To overcome these limitations, the authors propose Z-Perturbation Reinforcement Learning (ZPRL), which uniquely integrates the Variational Information Bottleneck (VIB) with flow matching. By freezing the main policy network and learning residual perturbations exclusively in a compact, task-relevant latent space, ZPRL enables lightweight, smooth, and task-aligned online adaptation. Evaluated across eight simulated and four real-world tasks, ZPRL substantially outperforms strong baselines, achieving a 33.7% average improvement in success rate on physical robots while demonstrating superior sample efficiency and more stable exploration behavior.

Abstract

Pretrained imitation policies have become a strong foundation for robot manipulation, but they often require online improvement to overcome execution errors, limited dataset coverage, and deployment mismatch. A central question is therefore how reinforcement learning (RL) should adapt policies after offline pretraining. Existing lightweight methods commonly apply residual corrections directly in action space, but this often leads to noisy and poorly structured exploration. In this work, we propose Z-Perturbation Reinforcement Learning (ZPRL), an approach that steers pretrained policies through a compact bottleneck latent rather than through policy weights or output actions. During offline training, we augment the policy with a plug-and-play variational information bottleneck (VIB) module to extract a task-relevant latent interface from observation embeddings. During online finetuning, the base policy is frozen and RL learns only a residual perturbation on this latent, whose decoded representation conditions the frozen action generator. We instantiate ZPRL on flow-matching policies and evaluate it on eight simulation tasks and four real-world tasks. Across diverse manipulation settings, ZPRL improves both sample efficiency and final performance over strong post-training baselines. In the real world, ZPRL improves the average success rate on four tasks by 33.7% over imitation base policies while producing smoother exploration behaviors than an action residual counterpart. These results suggest that a compact, task-aligned bottleneck latent provides an effective interface for online RL adaptation. More videos can be found at https://manutdmoon.github.io/ZPRL/.

Problem

• robot policy adaptation
• offline pretraining
• online fine-tuning
• action residuals
• deployment mismatch

Innovation

• bottleneck latent
• policy steering
• residual reinforcement learning
• variational information bottleneck
• flow-matching policies