Disentangling Foreground and Background for vision-Language Navigation via Online Augmentation

2025-10-01

AI Summary

In Vision-and-Language Navigation (VLN), entanglement of foreground and background information hinders agent generalization to unseen environments. To address this, we propose a foreground-background disentangled online feature enhancement framework. First, semantic-enhanced landmark identification explicitly separates foreground from background in visual observations. Second, we introduce a consensus-driven dynamic feature selection mechanism that performs adaptive representation optimization during navigation via a two-stage voting process—integrating multi-instruction alignment and position-aware cues. Notably, this is the first work to incorporate dynamic consensus guidance into VLN feature selection, fully end-to-end integrated with vision-language pretrained models. Our method achieves state-of-the-art generalization performance on REVERIE and R2R benchmarks, significantly outperforming prior baselines. Empirical results validate that foreground-background disentanglement is critical for robust, generalizable navigation.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Following language instructions, vision-language navigation (VLN) agents are tasked with navigating unseen environments. While augmenting multifaceted visual representations has propelled advancements in VLN, the significance of foreground and background in visual observations remains underexplored. Intuitively, foreground regions provide semantic cues, whereas the background encompasses spatial connectivity information. Inspired on this insight, we propose a Consensus-driven Online Feature Augmentation strategy (COFA) with alternative foreground and background features to facilitate the navigable generalization. Specifically, we first leverage semantically-enhanced landmark identification to disentangle foreground and background as candidate augmented features. Subsequently, a consensus-driven online augmentation strategy encourages the agent to consolidate two-stage voting results on feature preferences according to diverse instructions and navigational locations. Experiments on REVERIE and R2R demonstrate that our online foreground-background augmentation boosts the generalization of baseline and attains state-of-the-art performance.

Problem

• Disentangling foreground and background features for vision-language navigation agents
• Enhancing generalization through online augmentation of visual representations
• Addressing underexplored significance of semantic and spatial visual cues

Innovation

• Online augmentation strategy disentangles foreground and background features
• Semantically-enhanced landmark identification separates candidate augmented features
• Consensus-driven voting consolidates feature preferences across navigation stages