Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation

2025-11-17

AI Summary

To address the weak spatial reasoning, high computational overhead, and substantial inference latency of large language models (LLMs) in vision-language navigation (VLN), this paper proposes R3, a dual-process cognitive framework comprising: (1) a lightweight expert model (Runner) for efficient low-level navigation; (2) a multimodal LLM augmented with chain-of-thought reasoning (Ruminator) for deep semantic and spatial inference; and (3) a dynamic mode-switching regulator (Regulator) that orchestrates task execution. R3 is the first framework to achieve zero-shot synergistic integration of general-purpose reasoning and domain-specific expertise without compromising navigation accuracy—while markedly reducing computational cost. On the REVERIE benchmark, R3 achieves absolute improvements of +3.28% in Success-weighted by Path Length (SPL) and +3.30% in Room-Guided SPL (RGSPL), demonstrating significantly enhanced instruction-following capability, task completion rate, and robustness in complex 3D environments.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-and-Language Navigation (VLN) requires an agent to dynamically explore complex 3D environments following human instructions. Recent research underscores the potential of harnessing large language models (LLMs) for VLN, given their commonsense knowledge and general reasoning capabilities. Despite their strengths, a substantial gap in task completion performance persists between LLM-based approaches and domain experts, as LLMs inherently struggle to comprehend real-world spatial correlations precisely. Additionally, introducing LLMs is accompanied with substantial computational cost and inference latency. To address these issues, we propose a novel dual-process thinking framework dubbed R3, integrating LLMs' generalization capabilities with VLN-specific expertise in a zero-shot manner. The framework comprises three core modules: Runner, Ruminator, and Regulator. The Runner is a lightweight transformer-based expert model that ensures efficient and accurate navigation under regular circumstances. The Ruminator employs a powerful multimodal LLM as the backbone and adopts chain-of-thought (CoT) prompting to elicit structured reasoning. The Regulator monitors the navigation progress and controls the appropriate thinking mode according to three criteria, integrating Runner and Ruminator harmoniously. Experimental results illustrate that R3 significantly outperforms other state-of-the-art methods, exceeding 3.28% and 3.30% in SPL and RGSPL respectively on the REVERIE benchmark. This pronounced enhancement highlights the effectiveness of our method in handling challenging VLN tasks.

Problem

• Bridging performance gap between LLMs and experts in navigation tasks
• Reducing computational cost and latency of LLM-based navigation systems
• Improving spatial reasoning accuracy for vision-language navigation agents

Innovation

• Dual-process framework integrating LLMs with navigation expertise
• Lightweight transformer expert for efficient navigation execution
• Multimodal LLM with chain-of-thought reasoning for complex situations