OctoSense: Self-Supervised Learning for Multimodal Robot Perception

2026-06-25

AI Summary

This work addresses the challenges of modeling heterogeneous sensor modalities—characterized by divergent representations, asynchronous sampling rates, latency, and noise—in multimodal robotic perception, particularly under adverse conditions such as low-light environments or sensor degradation. To this end, the authors present an open-source multimodal platform integrating stereo RGB, event cameras, LiDAR, thermal imaging, IMU, RTK-GPS, and proprioceptive sensors, along with a late-fusion masked autoencoder framework. The approach employs modality-specific tokenizers tailored to each sensor’s spatiotemporal characteristics, leverages time-synchronized data collection and self-supervised pretraining, and enables efficient streaming inference via token caching. Evaluated on NVIDIA RTX 5090 and Jetson Orin NX platforms, the model achieves inference latencies of 6.68 ms and 112 ms, respectively, outperforming existing image foundation models in optical flow, depth estimation, semantic segmentation, and ego-motion tasks while demonstrating exceptional robustness in degraded sensing conditions.

Abstract

We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the dataset, code, and supplementary videos: https://abisulco.com/octosense/.

Problem

• multimodal perception
• sensor fusion
• self-supervised learning
• robotics
• degraded sensing

Innovation

• multimodal perception
• self-supervised learning
• masked autoencoder
• heterogeneous sensor fusion
• robotics foundation model