🤖 AI Summary
Autonomous agents struggle to effectively follow multimodal navigation instructions (e.g., language and vision) in dynamic environments. Method: This paper proposes “Virtual Guidance”—a mid-level, spatially grounded visual representation overlaid onto the camera view, bridging high-level semantic instructions and low-level perception. We design an end-to-end augmented reality (AR) navigation framework integrating multimodal instruction encoding, real-time AR rendering, reinforcement learning–based policy training, and sim-to-real domain adaptation. Contribution/Results: To our knowledge, this is the first work to explicitly transform non-visual navigation signals into interpretable, spatially aware visual guidance rendered in real time. The approach enhances both interpretability and execution robustness. Experiments across diverse simulated scenarios demonstrate significant improvements over non-visual baselines in navigation success rate, cross-platform adaptability, and resilience to environmental disturbances.
📝 Abstract
In the context of autonomous navigation, effectively conveying abstract navigational cues to agents in dynamic environments presents significant challenges, particularly when navigation information is derived from diverse modalities such as both vision and high-level language descriptions. To address this issue, we introduce a novel technique termed `Virtual Guidance,' which is designed to visually represent non-visual instructional signals. These visual cues are overlaid onto the agent's camera view and served as comprehensible navigational guidance signals. To validate the concept of virtual guidance, we propose a sim-to-real framework that enables the transfer of the trained policy from simulated environments to real world, ensuring the adaptability of virtual guidance in practical scenarios. We evaluate and compare the proposed method against a non-visual guidance baseline through detailed experiments in simulation. The experimental results demonstrate that the proposed virtual guidance approach outperforms the baseline methods across multiple scenarios and offers clear evidence of its effectiveness in autonomous navigation tasks.
