P$^{3}$Nav: End-to-End Perception, Prediction and Planning for Vision-and-Language Navigation

2026-03-18

AI Summary

This work proposes the first end-to-end unified framework for vision-and-language navigation that overcomes the limitations of existing methods, which often fail to achieve effective path planning due to insufficient scene understanding. By deeply integrating perception, prediction, and planning, the framework enhances environmental comprehension through multi-granularity visual perception, models latent states via future waypoint prediction, and constructs semantic maps based on anticipated waypoints to enable proactive and forward-looking decision-making. This approach departs from conventional passive planning paradigms that rely heavily on historical context. Extensive experiments demonstrate state-of-the-art performance across multiple benchmarks—REVERIE, R2R-CE, and RxR-CE—with significant improvements in navigation success rates.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In Vision-and-Language Navigation (VLN), an agent is required to plan a path to the target specified by the language instruction, using its visual observations. Consequently, prevailing VLN methods primarily focus on building powerful planners through visual-textual alignment. However, these approaches often bypass the imperative of comprehensive scene understanding prior to planning, leaving the agent with insufficient perception or prediction capabilities. Thus, we propose P$^{3}$Nav, a novel end-to-end framework integrating perception, prediction, and planning in a unified pipeline to strengthen the VLN agent's scene understanding and boost navigation success. Specifically, P$^{3}$Nav augments perception by extracting complementary cues from object-level and map-level perspectives. Subsequently, our P$^{3}$Nav predicts waypoints to model the agent's potential future states, endowing the agent with intrinsic awareness of candidate positions during navigation. Conditioned on these future waypoints, P$^{3}$Nav further forecasts semantic map cues, enabling proactive planning and reducing the strict reliance on purely historical context. Integrating these perceptual and predictive cues, a holistic planning module finally carries out the VLN tasks. Extensive experiments demonstrate that our P$^{3}$Nav achieves new state-of-the-art performance on the REVERIE, R2R-CE, and RxR-CE benchmarks.

Problem

• Vision-and-Language Navigation
• scene understanding
• perception
• prediction
• planning

Innovation

• end-to-end navigation
• perception-prediction-planning integration
• waypoint prediction
• semantic map forecasting
• vision-and-language navigation