CrossMaps: Confidence-Aware Open-Vocabulary Semantic Mapping for Rover Navigation

2026-06-15

AI Summary

This work addresses the challenge of inefficient semantic mapping for Mars rover navigation in partially observable environments by introducing CrossMaps—a real-time, confidence-aware, open-vocabulary semantic mapping approach. CrossMaps integrates RGB-D data with multi-scale CLIP embeddings and jointly optimizes noisy observations through geometric, semantic, and temporal confidence cues. It further incorporates a dual short-term and long-term memory mechanism to convert high-confidence semantic units into persistent landmarks. Built upon the VLMaps framework and deployed on a Jetson Orin platform, the system generates semantic heatmaps in real time that support natural language queries, significantly enhancing the rover’s semantic perception and autonomous navigation capabilities in complex environments.

Abstract

Rovers rely on perception to maintain spatial maps that encode both objects and sensor quality (e.g., range reliability, lighting artifacts, data density), guiding data fusion, embedding updates, and navigation under partial observability. To study these coupled perception-navigation processes, we present CrossMaps, a real-time confidence-aware open-vocabulary semantic mapping pipeline that constructs language-queryable maps from RGB-D data. Building on VLMaps-style approaches, CrossMaps integrates multi-scale CLIP embeddings with confidence-aware fusion and a dual-memory architecture consisting of Short-Term Memory (STM) and Long-Term Memory (LTM). The STM aggregates noisy visual observations using geometric, semantic, and temporal confidence cues, while confident and coherent cells are promoted to the LTM as persistent semantic landmarks. Designed for deployment with a Jetson Orin-powered UGV alongside SLAM, CrossMaps runs in real time and produces semantic heatmaps that can be queried with natural language to guide rover navigation.

Problem

• semantic mapping
• rover navigation
• open-vocabulary
• confidence-aware
• partial observability

Innovation

• confidence-aware fusion
• open-vocabulary semantic mapping
• dual-memory architecture
• CLIP embeddings
• language-queryable maps