Slow Brain, Fast Planner: Latency-Resilient VLM-Augmented Urban Navigation

2026-06-18

AI Summary

This work addresses the challenge that existing learning-based path planners often fail to accurately assess the semantic plausibility of trajectories in complex urban environments, leading robots to enter restricted zones or adopt improper orientations. To overcome this limitation, the authors propose a training-free, latency-resilient trajectory fusion mechanism that leverages a vision-language model (VLM) to assign high-level semantic scores to candidate trajectories. By integrating these delayed VLM decisions—up to several seconds—into real-time planning through geometric similarity matching and an exponential decay weighting strategy, the method significantly enhances planning reliability. Evaluated across approximately 2,000 real-world scenarios, the approach reduces average displacement error by 30%. Simulations further demonstrate over 80% task success rates even with VLM delays of up to five seconds, with successful validation also achieved in campus-scale deployments.

Abstract

Learning-based planners for sidewalk navigation can generate diverse candidate trajectories in real time, yet their scoring functions often fail to select the best trajectory in challenging situations, outputting trajectories that make the mobile robot drive onto grass, toward pedestrians, or in the wrong direction, even when better candidates exist in the same set. We call this the trajectory scoring gap: in real-world sidewalk navigation, the gap between an anchor-based planner's top choice and the best possible candidate is substantial, likely due to limited high-level scene understanding capability of the planner. Rather than replacing the planner with an end-to-end Vision-Language-Action model, we propose a VLM-Planner interface that uses a VLM to select a candidate index from the planner's proposal set and then fuse it with the planner's initial output. However, VLMs take 1--3s per query and so cannot directly drive a 5--20Hz control loop. We contribute a training-free, latency-resilient trajectory-level fusion layer that turns a stale VLM selection into real-time planner scoring via geometric similarity with exponential decay. On $\sim$2,000 challenging real-world scenarios (e.g., junctions, pedestrian encounters), VLM selection achieves 30% ADE reduction versus the planner's best selection, while the planner remains competitive in routine situations. In simulation, Score Fusion maintains >80% success rate with delays up to 5s. We demonstrate the full system on a mobile robot navigating challenging campus sidewalks with varied network latency.

Problem

• trajectory scoring gap
• urban navigation
• mobile robot
• candidate trajectory selection
• scene understanding

Innovation

• VLM-Augmented Planning
• Latency-Resilient Fusion
• Trajectory Scoring Gap
• Geometric Similarity Matching
• Vision-Language Model