IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning

2025-05-15

AI Summary

In robot policy fine-tuning, conventional pipelines—imitation learning (IL) pretraining followed by reinforcement learning (RL) fine-tuning—suffer from unstable exploration, low sample efficiency, and performance collapse. This paper proposes an alternating IL/RL fine-tuning paradigm. A gradient orthogonal separation mechanism decouples imitation and reinforcement objectives, with theoretical analysis revealing its intrinsic benefits for stability and sample efficiency. We further design a gradient subspace decomposition module that enables plug-and-play integration of behavior cloning with standard RL algorithms (e.g., PPO, SAC). Evaluated on 14 robotic manipulation and locomotion tasks, our method achieves an average 3.8× improvement in sample efficiency (up to 6.3×) and boosts task success rate from 12% to 88% (+76 percentage points), while robustly suppressing performance collapse under both sparse/dense and short-/long-horizon reward settings.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Imitation learning (IL) and reinforcement learning (RL) each offer distinct advantages for robotics policy learning: IL provides stable learning from demonstrations, and RL promotes generalization through exploration. While existing robot learning approaches using IL-based pre-training followed by RL-based fine-tuning are promising, this two-step learning paradigm often suffers from instability and poor sample efficiency during the RL fine-tuning phase. In this work, we introduce IN-RIL, INterleaved Reinforcement learning and Imitation Learning, for policy fine-tuning, which periodically injects IL updates after multiple RL updates and hence can benefit from the stability of IL and the guidance of expert data for more efficient exploration throughout the entire fine-tuning process. Since IL and RL involve different optimization objectives, we develop gradient separation mechanisms to prevent destructive interference during ABBR fine-tuning, by separating possibly conflicting gradient updates in orthogonal subspaces. Furthermore, we conduct rigorous analysis, and our findings shed light on why interleaving IL with RL stabilizes learning and improves sample-efficiency. Extensive experiments on 14 robot manipulation and locomotion tasks across 3 benchmarks, including FurnitureBench, OpenAI Gym, and Robomimic, demonstrate that ABBR can significantly improve sample efficiency and mitigate performance collapse during online finetuning in both long- and short-horizon tasks with either sparse or dense rewards. IN-RIL, as a general plug-in compatible with various state-of-the-art RL algorithms, can significantly improve RL fine-tuning, e.g., from 12% to 88% with 6.3x improvement in the success rate on Robomimic Transport. Project page: https://github.com/ucd-dare/IN-RIL.

Problem

• Combines IL and RL for stable policy fine-tuning
• Prevents gradient interference in IL-RL optimization
• Improves sample efficiency in robot learning tasks

Innovation

• Interleaves IL and RL for stable fine-tuning
• Uses gradient separation to prevent conflicts
• Improves sample efficiency across diverse tasks