GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model

2026-07-07

AI Summary

This work addresses the limitations of traditional vision-based navigation—namely, low data efficiency and deployment complexity due to reliance on specialized visual encoders and large-scale cross-embodiment datasets. The authors propose a novel paradigm that freezes a multimodal large language model (MLLM) and performs short-to-mid-range waypoint navigation solely through low-rank adaptation (LoRA) applied to the language tower. Navigation instructions and waypoints are unified into discrete token outputs, eliminating the need for additional visual encoders or continuous regression heads. This approach is the first to enable discrete-token navigation using only the language tower, augmented with a soft decoding auxiliary loss to recover metric structure. Trained on just 8.7 hours of open-source data, the method achieves zero-shot transfer to four unseen real-world environments, attaining target arrival errors of 0.25–0.42 meters across 20 physical trials.

Abstract

Visual navigation policies built on large pretrained models have so far followed a common recipe: a dedicated visual encoder, a bespoke action head, and training on thousands of hours of cross-embodiment datasets. We ask whether this recipe is necessary. In this paper, we introduce GemNav, a visual robot navigation policy that adapts a frozen Multimodal Large Language Model (MLLM) for short-to-medium horizon waypoint navigation using Low-Rank Adaptation (LoRA) on the language tower alone, with no auxiliary visual encoder and no continuous regression head. Waypoints and categorical navigation signals share a single discrete token vocabulary generated by the language-model head, and a soft-decoded auxiliary loss recovers the metric structure that pure cross-entropy training discards. On a single 8.7-hour open corpus, roughly three orders of magnitude smaller than competing training sets, the policy transfers zero-shot to four physically distinct unseen environments and stops within 0.25-0.42m of the goal across 20 real-world trials covering an open carpark, an obstacle carpark, a long outdoor chemical yard, and an indoor warehouse. Conditioning on short image histories improves offline metrics but yields no robot benefit, pointing to a ceiling on what temporal context adds once pretrained vision features are in place. These results indicate that discrete-token adaptation of frozen MLLMs can provide a data-efficient, deployable alternative for foundation model robot navigation.

Problem

• visual robot navigation
• multimodal large language model
• discrete-token representation
• data-efficient learning
• foundation model

Innovation

• discrete-token navigation
• frozen MLLM adaptation
• LoRA
• zero-shot transfer
• data-efficient robot navigation