Traj-VLN: Learning Pixel-Space Interaction via Autoregressive Trajectory Generation

2026-07-12

AI Summary

This work addresses the challenge of modeling spatial interactions in vision-and-language navigation (VLN) caused by the absence of depth or explicit 3D geometric priors. The authors propose a novel paradigm that eschews explicit 3D information by decomposing navigation into subtasks and autoregressively generating trajectory coordinates directly in 2D pixel space, thereby establishing pixel-level alignment between language instructions and visual observations. Built upon a vision-language model, the approach predicts continuous paths conditioned on historical observations and linguistic guidance, substantially reducing reliance on computational resources and large-scale training data. Experimental results demonstrate that this method achieves state-of-the-art performance in both cross-environment generalization and navigation accuracy, offering the first empirical validation of the effectiveness and efficiency of purely pixel-based trajectory supervision in VLN.

Abstract

Benefiting from the powerful priors embedded in large-scale pre-training data and the emerging commonsense reasoning ability, large language models (LLMs) have shown unprecedented generalization capabilities in many research fields. Recently, projecting visual embeddings into the language space via vision-language models (VLMs) to achieve sim-toreal and cross-scene generalization has become a prevailing paradigm in the field of Vision-and-Language Navigation in Continuous Environments (VLN-CE). VLN requires an embodied agent to navigate through unseen environments following natural linguistic instructions. We emphasize that a VLN task can be decomposed into a sequence of sub-tasks, each corresponding to a process of 3D spatial interaction with the environments described by instructions such as "walk to the end of the sofa and turn left." However, such spatial interactions involving moving into the image along the direction of depth sensing are puzzling for VLMs as they were predominantly trained on conversations with RGB images. Rather than incorporating depth or 3D geometric information-which VLMs rarely encounter during pretrainingwe propose an alternative approach: fine-tuning VLMs to learn navigation interactions directly in 2D pixel space through autoregressive trajectory generation. Given a linguistic instruction and historical observations, our model sequentially predicts a series of pixel coordinates, drawing a trajectory from the bottom center of the current observation. While prior work has proved that pixel-goal supervision outperforms learning of discrete actions, our experiments further verify that the supervision of pixel-space trajectory significantly enhances VLN performance. Moreover, we demonstrate that our flagship model achieves state-of-the-art level performance with relatively limited computational resources and training data.

Problem

• Vision-and-Language Navigation
• 3D spatial interaction
• visual language models
• depth perception
• continuous environments

Innovation

• autoregressive trajectory generation
• pixel-space interaction
• vision-language navigation
• 2D trajectory prediction
• VLN-CE