Nipping the Drift in the Bud: Retrospective Rectification for Robust Vision-Language Navigation

2026-02-06

AI Summary

This work addresses the challenge of exposure bias in vision-and-language navigation (VLN), where agents often deviate from intended paths, leading to semantic misalignment between instructions and environmental states that hinders recovery. To tackle this instruction-state misalignment—a problem newly identified in this study—we propose BudVLN, a novel framework that synthesizes semantically consistent corrective trajectories by rolling back the policy online and re-anchoring via counterfactual reasoning from valid historical states. The approach further incorporates geodesic oracle trajectory generation and decision-conditioned supervision to construct state-aligned learning signals. Evaluated on the R2R-CE and RxR-CE benchmarks, BudVLN achieves state-of-the-art success rates and SPL, significantly mitigating distributional shift and improving robustness in complex navigation tasks.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Vision-Language Navigation (VLN) requires embodied agents to interpret natural language instructions and navigate through complex continuous 3D environments. However, the dominant imitation learning paradigm suffers from exposure bias, where minor deviations during inference lead to compounding errors. While DAgger-style approaches attempt to mitigate this by correcting error states, we identify a critical limitation: Instruction-State Misalignment. Forcing an agent to learn recovery actions from off-track states often creates supervision signals that semantically conflict with the original instruction. In response to these challenges, we introduce BudVLN, an online framework that learns from on-policy rollouts by constructing supervision to match the current state distribution. BudVLN performs retrospective rectification via counterfactual re-anchoring and decision-conditioned supervision synthesis, using a geodesic oracle to synthesize corrective trajectories that originate from valid historical states, ensuring semantic consistency. Experiments on the standard R2R-CE and RxR-CE benchmarks demonstrate that BudVLN consistently mitigates distribution shift and achieves state-of-the-art performance in both Success Rate and SPL.

Problem

• Vision-Language Navigation
• Exposure Bias
• Instruction-State Misalignment
• Distribution Shift
• Imitation Learning

Innovation

• Retrospective Rectification
• Instruction-State Misalignment
• Counterfactual Re-anchoring
• Decision-conditioned Supervision
• Geodesic Oracle