Green for Go, Red for No: Visual Grounding via Semantic Segmentation for VLA Navigation Policies

2026-07-06

AI Summary

This work addresses the susceptibility of existing vision-language navigation (VLA) approaches to visual distractions and scene ambiguity, which often degrade path-planning accuracy. The authors propose a lightweight, training-free semantic guidance mechanism that leverages SegFormer for real-time semantic segmentation, marking traversable and non-traversable regions in green and red, respectively, and integrating these visual cues into the OmniVLA framework. This integration supports guidance from either observation-only or observation-goal image pairs. The study presents the first systematic evaluation of visual guidance in VLA, demonstrating substantial improvements: on the Grand Tour dataset, it reduces average waypoint error by 27%–44%—particularly for long-horizon instructions—and shortens predicted trajectories by 30%, while also revealing an implicit trajectory-length regularization effect.

Abstract

Vision-language-action (VLA) models enable robot navigation from natural language and visual goals, but remain susceptible to perceptual distractions and ambiguous scene interpretations. This paper presents the first empirical evaluation of visual grounding for VLA navigation policies. We propose a real-time segmentation-based grounding method that highlights traversable areas in green and non-traversable areas in red using SegFormer. Two variants are evaluated: observation-only segmentation and joint observation-goal augmentation. Using OmniVLA on the Grand Tour dataset, we show that visual grounding reduces the mean waypoint error by 27-44% at the farthest waypoint, depending on the instruction length. The benefits are greater for long instructions than for short instructions, and grounding provides little improvement for image goals. Normalized error analysis indicates that grounding primarily acts as a trajectory length regularizer, reducing the predicted path length by 30% without improving per-unit-distance reasoning. Our results indicate that visual grounding offers a simple, computationally inexpensive method to improve VLA navigation without model retraining, although it cannot compensate for missing training signals in out-of-distribution instructions.

Problem

• visual grounding
• VLA navigation
• perceptual distractions
• ambiguous scene interpretations
• semantic segmentation

Innovation

• visual grounding
• semantic segmentation
• VLA navigation
• SegFormer
• trajectory regularization