Problem
Research questions and friction points this paper is trying to address.
Vision-Language Navigation
on-policy exploration
semantic mismatch
hindsight instruction
trajectory-instruction alignment
Path-level Hindsight Instructions for Semantic Exploration in Vision-Language Navigation
🤖 AI Summary
This work addresses the semantic mismatch between agent trajectories and language instructions in vision-and-language navigation caused by on-policy exploration. The authors propose Phi-Nav, a framework employing a three-stage dual-supervision loop: it first conducts on-policy exploration guided by an oracle; then introduces a path-level hindsight instruction generation mechanism to convert semantically unlabeled exploration trajectories into dense supervision signals; and finally performs a second round of imitation learning using the synthesized trajectory-instruction pairs to achieve semantic alignment. Phi-Nav is the first to incorporate path-level hindsight instructions into navigation training, achieving competitive performance on the R2R-CE and RxR-CE benchmarks with only limited expert demonstration data, effectively bridging the semantic supervision gap inherent in on-policy methods.
📝 Abstract
On-policy exploration is a crucial component for training robust Vision-Language Navigation agents, as it exposes the policy to a broader state distribution. However, such exploration inevitably leads to trajectories that deviate from expert demonstrations, resulting in a semantic mismatch between the executed visual stream and the original language instruction. In this work, we address this challenge by introducing Phi-Nav, a unified on-policy framework that leverages hindsight reasoning to align instructions with the agent's actual exploratory journey. Specifically, Phi-Nav operates through a three-stage dual-supervision cycle: 1) the agent performs oracle-guided on-policy exploration, sampling a trajectory while learning from expert action feedback, 2) a hindsight speaker synthesizes a path-level hindsight instruction grounded in the collected visual observations, and 3) the agent conducts a second imitation pass, treating the synthesized trajectory-instruction pair as an additional expert demonstration. Through this process, Phi-Nav bridges the critical semantic supervision gap inherent in on-policy methods, transforming semantically unlabeled movement into dense training signals. Evaluations on the R2R-CE and RxR-CE benchmarks show that Phi-Nav yields competitive performance while requiring only a fraction of the expert demonstrations used by current baselines. These results underscore the necessity of semantic exploration in VLN, positioning Phi-Nav as an effective solution for training embodied agents with limited data.
Innovation
Methods, ideas, or system contributions that make the work stand out.
hindsight instruction
vision-language navigation
on-policy exploration
semantic alignment
dual-supervision cycle
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
