LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation

2026-06-26

AI Summary

This work addresses the challenge of deploying large foundation models for object-goal navigation on resource-constrained robotic platforms, where high computational demands hinder low-latency inference. To overcome this limitation, the authors propose a novel framework that integrates scene-graph-guided knowledge distillation with an E-RLVR reinforcement learning optimization strategy, effectively transferring advanced spatial-semantic reasoning capabilities from large models to a lightweight 4B-parameter vision-language model. The approach further incorporates token generation regularization and model quantization to enhance efficiency. Evaluated on the HM3D OVON benchmark, the method achieves a success rate of 34.5% while reducing inference latency by 82.8% and output overhead by 72.1%, marking the first demonstration of near-cloud-level navigation performance on an embedded GPU.

Abstract

Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context, serving as a catalyst for open-vocabulary tasks like ObjectNav. Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots. To address this challenge, we present a distillation strategy that transfers complex spatial-semantic reasoning from large frontier models into a lightweight, 4B-parameter local VLM for edge execution on embedded GPU devices (e.g., Jetson Orin). We first establish a State of the Art (SotA), Scene Graph (SG)-based pipeline using Claude Sonnet 4.6, achieving a 39.7% Success Rate (SR) on the HM3D OVON benchmark. We then demonstrate that fine-tuning Qwen3.5-4B on just 500 frontier reasoning traces effectively enables navigation capabilities, yielding a SR of 34.5%, narrowing the gap to the performance of large cloud models. Finally, we introduce E-RLVR with Token Generation (TG) regularization to compress output sequence lengths for physical deployment while grounding the agent in its task. This downstream optimization reduces TG overhead by 72.1% and latency by 71.8%. Combined with quantization, this joint strategy yields a cumulative 82.8% reduction in overall inference latency without significantly sacrificing performance, presenting a viable paradigm for local, low-latency VLM execution on mobile robots.

Problem

• Object Goal Navigation
• Vision Language Models
• On-Device Inference
• Low-Latency Deployment
• Embodied AI

Innovation

• Vision Language Models
• Model Distillation
• On-Device Inference
• Object Goal Navigation
• Token Generation Regularization