Structured Observation Language for Efficient and Generalizable Vision-Language Navigation

2026-03-29

AI Summary

This work addresses the limited generalization of existing vision-and-language navigation methods under environmental variations such as lighting and texture, which often rely heavily on large-scale visual pretraining. The authors propose a novel approach that partitions first-person RGB-D observations into a grid, extracting semantic, color, and depth information from each cell and converting it into structured textual descriptions. These descriptions are then concatenated with natural language navigation instructions and fed into a pretrained language model to make navigational decisions. By encoding visual inputs as structured language rather than learning end-to-end visual features, the method significantly enhances cross-environment generalization. Experiments demonstrate superior performance on standard benchmarks including R2R and RxR, as well as in real-world settings, while requiring fewer model parameters and less training data.

Abstract

Vision-Language Navigation (VLN) requires an embodied agent to navigate complex environments by following natural language instructions, which typically demands tight fusion of visual and language modalities. Existing VLN methods often convert raw images into visual tokens or implicit features, requiring large-scale visual pre-training and suffering from poor generalization under environmental variations (e.g., lighting, texture). To address these issues, we propose SOL-Nav (Structured Observation Language for Navigation), a novel framework that translates egocentric visual observations into compact structured language descriptions for efficient and generalizable navigation. Specifically, we divide RGB-D images into a N*N grid, extract representative semantic, color, and depth information for each grid cell to form structured text, and concatenate this with the language instruction as pure language input to a pre-trained language model (PLM). Experimental results on standard VLN benchmarks (R2R, RxR) and real-world deployments demonstrate that SOL-Nav significantly reduces the model size and training data dependency, fully leverages the reasoning and representation capabilities of PLMs, and achieves strong generalization to unseen environments.

Problem

• Vision-Language Navigation
• generalization
• visual pre-training
• environmental variations

Innovation

• structured language representation
• vision-language navigation
• pre-trained language models
• environment generalization
• egocentric observation encoding