PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model

2026-07-20

AI Summary

This work addresses key challenges in vision-and-language navigation (VLN)—namely, cross-modal alignment, temporal compression, and action grounding—by proposing PGN, an offline VLN system. Built upon a frozen EVA-ViT-G/14 visual encoder and the OpenPangu-7B language model, PGN employs a Q-Former and MLP projector to achieve effective vision–language alignment, while limiting fine-tuning to structured token embeddings and LoRA adapters for efficient transfer. A novel five-frame temporal sampling strategy and a tailored reasoning-to-action output format are introduced to enhance alignment with action sequences. Evaluated on 500 expert trajectories in an open-loop setting, the system achieves a normalized action matching (NAM) rate of 62.29% and a non-empty rate (NER) of 100%, demonstrating its effectiveness.

Abstract

Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural-language instruction and predict actions from temporally ordered visual observations. Adapting a multimodal large language model to VLN requires visual-language alignment, compact temporal inputs, action-space grounding, and stable training on the target hardware. This technical report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B. Training proceeds in two stages. First, PGMM aligns a frozen EVA-ViT-G/14 vision encoder with the frozen language backbone by training a Q-Former and a two-layer MLP projector. Second, PGN adapts the aligned model to expert navigation trajectories using five-observation windows, epoch-dependent temporal sampling, and a reasoning-then-action output format; this stage freezes the aligned visual pathway and updates three structural-token embeddings and LoRA adapters. The implementation combines mixed-precision computation, selective FP32 computation, and DeepSpeed ZeRO-2 on eight Ascend 910B NPUs. Under teacher-forced, open-loop evaluation on 500 held-out expert trajectories, V9 reports a 62.29% Normalized Action Match (NAM) and a 100.00% Non-empty Rate (NER). These metrics quantify offline expert-action alignment rather than closed-loop navigation success; evaluating error accumulation, path efficiency, and goal completion remains future work.

Problem

• Vision-Language Navigation
• multimodal foundation model
• action prediction
• visual-language alignment
• embodied agent

Innovation

• Vision-Language Navigation
• Multimodal Foundation Model
• LoRA Adaptation
• Temporal Sampling
• Hardware-Efficient Training