Do Visual Imaginations Improve Vision-and-Language Navigation Agents?

2025-03-20

AI Summary

Can visual imagination improve the generalization capability of vision-language navigation (VLN) agents in unseen environments? This paper proposes the first method that explicitly integrates text-to-image diffusion-generated visual imagination as multimodal navigational cues. Our approach segments navigation instructions to identify key visual concepts—particularly landmarks—and employs a diffusion model to synthesize corresponding landmark images. To strengthen cross-modal alignment, we introduce a referential expression–image matching auxiliary loss that explicitly supervises language–vision grounding. The method significantly enhances the embodied grounding capacity of VLN agents, achieving approximately +1.0% absolute improvement in success rate and up to +0.5% in SPL across multiple benchmarks—including R2R and CVDN. These results empirically validate that generative visual imagination effectively mitigates the perceptual limitations of purely language-driven navigation, establishing a novel paradigm for multimodal representation learning in embodied intelligence.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Vision-and-Language Navigation (VLN) agents are tasked with navigating an unseen environment using natural language instructions. In this work, we study if visual representations of sub-goals implied by the instructions can serve as navigational cues and lead to increased navigation performance. To synthesize these visual representations or imaginations, we leverage a text-to-image diffusion model on landmark references contained in segmented instructions. These imaginations are provided to VLN agents as an added modality to act as landmark cues and an auxiliary loss is added to explicitly encourage relating these with their corresponding referring expressions. Our findings reveal an increase in success rate (SR) of around 1 point and up to 0.5 points in success scaled by inverse path length (SPL) across agents. These results suggest that the proposed approach reinforces visual understanding compared to relying on language instructions alone. Code and data for our work can be found at https://www.akhilperincherry.com/VLN-Imagine-website/.

Problem

• Enhancing VLN agents with visual imaginations
• Using text-to-image models for navigational cues
• Improving navigation success rates with visual aids

Innovation

• Text-to-image diffusion model for visual imaginations
• Added modality for landmark cues in navigation
• Auxiliary loss to relate imaginations with instructions