Joint On-and-Off Policy Learning for Vision-and-Language Navigation

2026-07-15

AI Summary

This work addresses the challenge in Vision-and-Language Navigation (VLN) of simultaneously achieving the error-recovery capability of imitation learning and the exploration efficiency of reinforcement learning. The authors propose JOP-VLN, a novel framework that deeply integrates off-policy imitation learning with on-policy reinforcement learning for the first time in VLN. Through a three-stage training pipeline, the method first pretrains a base policy using expert demonstrations, then generates high-entropy exploration trajectories via DAgger augmented with an error-correction-prioritized trajectory ranking mechanism, and finally performs joint policy optimization. This approach substantially improves both training efficiency and robustness, achieving state-of-the-art success rates of 69.9% on the VLN-CE R2R benchmark and 68.0% on RxR, with the R2R result setting a new record.

Abstract

Vision-and-Language Navigation (VLN) necessitates an embodied agent to navigate in the physical world by adhering to natural language instructions. Recent advancements in Vision-Language Models (VLM) have propelled the development of VLM-based VLN methods with two predominant paradigms: (1) imitation learning (IL) on expert demonstrations, followed by the Dataset Aggregation (DAgger) algorithm to bolster error recovery capabilities; (2) reinforcement learning (RL) driven by verifiable rewards to enhance reasoning and exploration. A notable gap is the absence of integration between these two distinct paradigms. This paper introduces JOP-VLN, a novel VLN framework that synergistically combines off-policy imitation learning and on-policy exploration within a three-stage training pipeline. Initially, IL is employed on expert demonstrations to acquire basic navigation skills. Subsequently, the DAgger algorithm is utilized to generate heuristic exploration trajectories, which are then used for imitation learning to improve error recovery capabilities. Finally, a joint on-and-off policy learning framework is implemented, featuring high-entropy trajectory sampling to enhance RL training efficiency and an error-correction-prioritized trajectory sorting strategy for effective error correction. Extensive experiments demonstrate the efficacy of JOP-VLN, achieving success rates of 69.9% and 68.0% on the VLN-CE R2R and RxR benchmarks, respectively, setting a new state-of-the-art on R2R. Project page: https://qingrongh.github.io/JOP-VLN.

Problem

• Vision-and-Language Navigation
• Imitation Learning
• Reinforcement Learning
• On-policy Learning
• Off-policy Learning

Innovation

• joint on-and-off policy learning
• vision-and-language navigation
• imitation learning
• reinforcement learning
• trajectory sampling