Turning Adaptation into Assets: Cross-Domain Bridging for Online Vision-Language Navigation

2026-05-22

AI Summary

This work addresses the challenges of catastrophic forgetting and negative transfer faced by vision-and-language navigation agents when adapting online in non-stationary environments. To this end, the authors propose IDEA, a novel framework that reframes adaptation as the accumulation and composition of dynamic knowledge assets. IDEA innovatively integrates domain-coordinate embeddings, Fisher information–weighted soft prompt optimization, and a convex-hull projection–based cross-domain bridging mechanism to construct a dynamic asset repository that enables efficient training-free adaptation. Evaluated on the REVERIE, R2R, and R2R-CE benchmarks, IDEA substantially outperforms existing approaches, achieving state-of-the-art performance in training-free cross-domain adaptation.

Abstract

Navigating under non-stationary environment shifts poses a critical challenge for a Vision-and-Language Navigation (VLN) agent deployed in the wild. Yet, existing Test-Time Adaptation (TTA) methods for VLN largely treat online adaptation as transient, isolated updates, leading to catastrophic forgetting and negative transfer. To overcome these issues, we propose Inter-Domain BridgE with Historical Assets (IDEA), a novel TTA framework that transforms adaptation into the accumulation and composition of assets. Specifically, IDEA introduces soft prompts optimized via a Fisher-guided weighting scheme to capture the transferable knowledge. These optimized prompts are then augmented with domain coordinates to form a dynamic asset library. Leveraging this library, IDEA constructs a cross-domain bridge by projecting the target domain onto the convex hull of historical knowledge. These designs form a complementary loop: the evolving library underpins bridge construction, while the bridge provides superior initialization to accelerate asset optimization. Extensive experiments across REVERIE, R2R, and R2R-CE benchmarks demonstrate the consistent superiority of IDEA over existing methods, showcasing its ability to enable training-free adaptation via asset sharing.

Problem

• Vision-and-Language Navigation
• Test-Time Adaptation
• Catastrophic Forgetting
• Negative Transfer
• Non-stationary Environment

Innovation

• Test-Time Adaptation
• Vision-and-Language Navigation
• Soft Prompts
• Cross-Domain Bridging
• Asset Accumulation