Do You Know the Way? Human-in-the-Loop Understanding for Fast Traversability Estimation in Mobile Robotics

2025-04-28

IEEE Robotics and Automation Letters

AI Summary

To address the challenge of real-time, robust traversability estimation for mobile robots in unstructured outdoor environments, this paper proposes the first foundation-model-based human-in-the-loop framework. Unlike conventional approaches relying on dense annotations or prior domain knowledge, our method integrates vision foundation models (CLIP/SAM variants), active learning, online Bayesian updating, and joint geometric-semantic representation. It enables on-deployment acquisition of sparse human annotations (only 5–10 queries) and supports few-shot incremental adaptation, effectively mitigating domain shift. Evaluated on both simulation and real-world robot datasets, the framework achieves state-of-the-art performance: ≥92% accuracy and <50 ms inference latency—without requiring large-scale pretraining datasets or robot motion priors.

Technology Category

• Intelligent Robots: State Estimation
• Humans and AI: Human-in-the-loop Machine Learning
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The increasing use of robots in unstructured environments necessitates the development of effective perception and navigation strategies to enable field robots to successfully perform their tasks. In particular, it is key for such robots to understand where in their environment they can and cannot travel -- a task known as traversability estimation. However, existing geometric approaches to traversability estimation may fail to capture nuanced representations of traversability, whereas vision-based approaches typically either involve manually annotating a large number of images or require robot experience. In addition, existing methods can struggle to address domain shifts as they typically do not learn during deployment. To this end, we propose a human-in-the-loop (HiL) method for traversability estimation that prompts a human for annotations as-needed. Our method uses a foundation model to enable rapid learning on new annotations and to provide accurate predictions even when trained on a small number of quickly-provided HiL annotations. We extensively validate our method in simulation and on real-world data, and demonstrate that it can provide state-of-the-art traversability prediction performance.

Problem

• Develops human-in-the-loop traversability estimation for robots
• Addresses domain shifts by learning during deployment
• Uses foundation models for rapid learning on few annotations

Innovation

• Human-in-the-loop annotations for traversability estimation
• Foundation model enables rapid learning on new annotations
• Accurate predictions with few quickly-provided human annotations