HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning

2026-06-22

AI Summary

Behavioral cloning in robotic manipulation is prone to error accumulation and distribution shift, hindering its robust deployment in industrial settings. This work proposes a plug-and-play residual reinforcement learning fine-tuning framework that learns a residual policy on top of vision-language-action (VLA) model outputs, augmented with a human-in-the-loop mechanism to enable online correction of suboptimal actions and safe, efficient exploration. The approach introduces the first model-agnostic residual fine-tuning architecture, compatible with diverse VLA models, and achieves an average task success rate exceeding 95% after only 1.5 hours of real-robot online training. This significantly enhances the real-world deployability of behavioral cloning policies while maintaining compatibility with existing foundation models.

Abstract

Recent advancements in generative imitation learning have significantly propelled the field of robotic manipulation. However, the majority of existing models rely heavily on Behavior Cloning (BC), a paradigm that suffers from compounding errors and distributional shift. Consequently, the efficacy of these models in practical industrial deployments remains limited. To address these challenges, we introduce a novel, plug-and-play fine-tuning pipeline designed to facilitate the robust deployment of Vision-Language-Action (VLA) models in real-world environments. In contrast to contemporary reinforcement learning (RL) fine-tuning strategies, which are often constrained by specific model architectures, our proposed framework is model-agnostic and adaptable to a diverse range of VLA models. We conceptualize VLA-generated actions as a unified interface, upon which we train a residual policy. This policy is designed to rectify suboptimal actions and address the distributional shift inherent in imitation learning. Additionally, we incorporate human-in-the-loop guidance to ensure safe exploration and maximize training efficiency. We conduct experiments directly in real-world robotic settings. The results demonstrate that within only 1.5 hour of real-world online RL training, the average success rate exceeds 95% on real robots. Our work presents a practical solution for deploying behavior cloning models in industrial scenarios.

Problem

• behavior cloning
• distributional shift
• robotic manipulation
• real-world deployment
• compounding errors

Innovation

• model-agnostic
• residual reinforcement learning
• human-in-the-loop
• Vision-Language-Action (VLA)
• real-world robotic fine-tuning