Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments

2025-06-17

AI Summary

Large Vision-Language Models (VLMs) suffer from high computational overhead and weak modeling of continuous motion signals, leading to insufficient real-time performance and control precision for human-centered visual navigation. To address this, we propose a Barlow Twins–based self-supervised learning framework that introduces an implicit language reasoning embedding mechanism—novelly encoding social cues and human intent directly into the visual feature latent space, enabling token-free end-to-end inference. Our method jointly processes RGB observations, motion instructions, and scene text via a redundancy-suppressed visual encoder, mapping raw inputs directly to short-horizon point-goal navigation commands. On offline unseen datasets and real-world experiments, our approach improves navigation success rates by 52.94% and 41.67%, respectively. Attention visualization further confirms superior localization accuracy for critical navigational elements.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large Vision-Language Models (VLMs) have demonstrated potential in enhancing mobile robot navigation in human-centric environments by understanding contextual cues, human intentions, and social dynamics while exhibiting reasoning capabilities. However, their computational complexity and limited sensitivity to continuous numerical data impede real-time performance and precise motion control. To this end, we propose Narrate2Nav, a novel real-time vision-action model that leverages a novel self-supervised learning framework based on the Barlow Twins redundancy reduction loss to embed implicit natural language reasoning, social cues, and human intentions within a visual encoder-enabling reasoning in the model's latent space rather than token space. The model combines RGB inputs, motion commands, and textual signals of scene context during training to bridge from robot observations to low-level motion commands for short-horizon point-goal navigation during deployment. Extensive evaluation of Narrate2Nav across various challenging scenarios in both offline unseen dataset and real-world experiments demonstrates an overall improvement of 52.94 percent and 41.67 percent, respectively, over the next best baseline. Additionally, qualitative comparative analysis of Narrate2Nav's visual encoder attention map against four other baselines demonstrates enhanced attention to navigation-critical scene elements, underscoring its effectiveness in human-centric navigation tasks.

Problem

• Enhancing real-time robot navigation using visual-language reasoning
• Reducing computational complexity for precise motion control
• Improving attention to navigation-critical elements in human-centric environments

Innovation

• Self-supervised learning with Barlow Twins loss
• Latent space reasoning for human intentions
• RGB-motion-text fusion for real-time navigation