LookasideVLN: Direction-Aware Aerial Vision-and-Language Navigation

2026-04-18

AI Summary

Existing aerial vision-language navigation methods overly rely on landmark descriptions while neglecting directional cues, resulting in shallow spatial understanding and high computational overhead. This work proposes a novel paradigm that explicitly models directional information inherent in natural language instructions by integrating egocentric lateral gaze (ELG), a spatial landmark knowledge base (SLKB), and a multimodal large language model (MLLM)-based navigation agent. The approach leverages a dynamic graph structure coupled with a lightweight memory mechanism to align multimodal inputs effectively. Evaluated under a single-step lookahead setting, the proposed method significantly outperforms the current state-of-the-art CityNavAgent, demonstrating the efficiency and critical role of directional cues in aerial vision-language navigation.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Aerial Vision-and-Language Navigation (Aerial VLN) enables unmanned aerial vehicles (UAVs) to follow natural language instructions and navigate complex urban environments. While recent advances have achieved progress through large-scale memory graphs and lookahead path planning, they remain limited by shallow instruction understanding and high computational cost. In particular, existing methods rely primarily on landmark descriptions, overlooking directional cues "a key source of spatial context in human navigation". In this work, we propose LookasideVLN, a new paradigm that exploits directional cues in natural language to achieve both more accurate spatial reasoning and greater computational efficiency. LookasideVLN comprises three core components: (1) an Egocentric Lookaside Graph (ELG) that dynamically encodes instruction-relevant landmarks and their directional relationships, (2) a Spatial Landmark Knowledge Base (SLKB) that provides lightweight memory retrieval from prior navigation experiences, and (3) a Lookaside MLLM Navigation Agent that aligns multimodal information from user instructions, visual observations, and landmark-direction information from ELG for path planning. Extensive experiments show that LookasideVLN significantly outperforms the state-of-the-art CityNavAgent, even with a single-level lookahead, demonstrating that leveraging directional cues is a powerful yet efficient strategy for Aerial VLN.

Problem

• Aerial Vision-and-Language Navigation
• directional cues
• spatial reasoning
• instruction understanding
• computational efficiency

Innovation

• directional cues
• Egocentric Lookaside Graph
• Spatial Landmark Knowledge Base
• Aerial Vision-and-Language Navigation
• multimodal navigation agent