AnyTraverse: An off-road traversability framework with VLM and human operator in the loop

2025-06-20

AI Summary

To address poor adaptability and cross-platform generalization in robot traversability segmentation for unstructured off-road environments, this paper proposes a language-driven, sparsely human-in-the-loop zero-shot real-time segmentation framework. Methodologically, it integrates vision-language models (VLMs) with natural language prompt engineering; human intervention is triggered only upon encountering unknown scenes or out-of-prompt categories, enabling dynamic adaptation without fine-tuning, new data collection, or platform-specific customization—ensuring plug-and-play deployment on arbitrary robotic platforms. Evaluated on multi-source outdoor datasets (RELLIS-3D, Freiburg Forest, RUGD) and real-world robotic platforms, the framework surpasses GA-NAV and Off-seg in traversable region identification accuracy while significantly reducing human supervision frequency. It introduces the first “language-driven + sparse human-robot collaboration” paradigm, offering a lightweight, generalizable, and scalable solution for野外 autonomous navigation.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Off-road traversability segmentation enables autonomous navigation with applications in search-and-rescue, military operations, wildlife exploration, and agriculture. Current frameworks struggle due to significant variations in unstructured environments and uncertain scene changes, and are not adaptive to be used for different robot types. We present AnyTraverse, a framework combining natural language-based prompts with human-operator assistance to determine navigable regions for diverse robotic vehicles. The system segments scenes for a given set of prompts and calls the operator only when encountering previously unexplored scenery or unknown class not part of the prompt in its region-of-interest, thus reducing active supervision load while adapting to varying outdoor scenes. Our zero-shot learning approach eliminates the need for extensive data collection or retraining. Our experimental validation includes testing on RELLIS-3D, Freiburg Forest, and RUGD datasets and demonstrate real-world deployment on multiple robot platforms. The results show that AnyTraverse performs better than GA-NAV and Off-seg while offering a vehicle-agnostic approach to off-road traversability that balances automation with targeted human supervision.

Problem

• Segments off-road traversability for autonomous navigation in unstructured environments
• Adapts to diverse robotic vehicles without extensive retraining
• Balances automation with human supervision for unexplored scenes

Innovation

• Combines VLM and human-operator for traversability
• Zero-shot learning avoids data collection
• Vehicle-agnostic approach balances automation