Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation

2026-05-26

AI Summary

This work addresses the zero-shot mapping of natural language instructions to spatial action sequences for multi-platform embodied navigation in unseen environments. It formulates embodied navigation as a unified language–vision–action translation task and introduces a framework that integrates pretrained multimodal large language models (MLLMs) with structured memory management (TDM) and a self-correction backtracking mechanism (SCB). Without any task-specific training, the approach generalizes effectively across four distinct navigation tasks and four heterogeneous robotic platforms. Evaluated on six benchmarks—including VLN-CE R2R, RxR, and HM3D-v2—it achieves success rates ranging from 40.0% to 77.7%, rivaling or even surpassing state-of-the-art navigation foundation models that rely on millions of expert trajectories for training.

Abstract

Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen. The dominant approach has been to scale vision-language-action (VLA) foundation models on ever-larger collections of robot trajectories. This paper argues that, for navigation specifically, generality can be obtained structurally, not only through data scale. The underlying decision structure of navigation reduces to a single Language-Vision-Robot Actions Translation. The language action emits semantic-level directional command and the vision action emits a pixel-level visual target. Both outputs lie inside the natural output manifold of pretrained multimodal large language models (MLLMs), so the task can be reasoned about by an agent rather than learned from robot data. Therefore, we present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families (VLN-CE, ObjectNav, EQA, and Aerial-VLN) and to four heterogeneous real robots (Wheeled, Quadruped, Humanoid robot, and a self-built UAV) in a zero-shot manner. Two agent-loop mechanisms make this unification practical. TODO List Memory (TDM) rewrites a structured checklist of pending sub-goals at every step, reciting the unfinished items back into the agent's most recent attention window. Second Chance Backtrack (SCB) rolls the robot back to the pre-error state and conditions the agent's next plan on the failed sub-trajectory, turning single-pass navigation into a self-correcting process. With zero training effort, Uni-LaViRA reaches 60.7% SR on VLN-CE R2R, 51.3% on VLN-CE RxR, 77.7% on HM3D-v2, 60.0% on HM3D-OVON, 54.7% on MP3D-EQA, and 40.0% on OpenUAV, matching or even surpassing recent training navigation foundation models that consume millions of samples and thousands of GPU-hours.

Problem

• Embodied Navigation
• Language-Vision-Action Translation
• Zero-shot Generalization
• Multimodal Large Language Models
• Heterogeneous Robots

Innovation

• Language-Vision-Robot Actions Translation
• Zero-shot Embodied Navigation
• Multimodal Large Language Models
• Self-correcting Navigation
• Unified Agentic Architecture