MAMMOTH: A Multi-Modal End-to-End Policy for Off-Road Mobility Robust to Missing Modality

2026-07-14

AI Summary

This work addresses the challenge of unreliable navigation for autonomous systems in unstructured off-road environments, where abrupt lighting changes, complex terrain, and sensor degradation severely degrade performance—particularly in existing approaches that overly rely on RGB inputs and fail under missing modalities. To overcome these limitations, we propose an end-to-end multimodal navigation framework that fuses RGB, thermal imaging, 3D point clouds, and ego-motion cues. Our approach uniquely integrates random modality dropout during training with a diffusion-based policy, enabling robust inference under arbitrary modality absence at test time. Furthermore, we incorporate traversability-aware heuristics to guide the generation of physically feasible, safe, and smooth trajectories. Extensive experiments in diverse real-world off-road scenarios—including nighttime conditions—demonstrate significant improvements over state-of-the-art methods in obstacle avoidance, terrain-aware planning, and robustness to missing modalities.

Abstract

Reliable autonomous navigation in unstructured off-road environments remains a critical unsolved challenge due to extreme terrain diversity, drastic illumination variations and acute sensor degradation. Recent developments have approached the problem as a traversability costmap estimation or visual navigation task. However, many exhibit heavy reliance on RGB modality, leading to poor performance in varied illumination such as glares, shadows or low ambient light. Achieving robust generalization in such conditions requires integrating modalities that provide supplementary scene information. Such multi-modal methods suffer from a rigid dependency on the presence of near-perfect sensor inputs, leaving them unable to robustly handle sensor degradation or individual modality failure. To address these limitations, we introduce MAMMOTH (MAsking Multi-Modal inputs for Off-road Traversability Heuristic-informed navigation), a unified end-to-end navigation policy for robust off-road visual-goal-conditioned navigation and undirected exploration. Specifically, MAMMOTH efficiently fuses multi-modal observations (RGB, Thermal, 3D Pointcloud and Ego Velocity) and is trained with a modality dropout scheme, enabling it to generalize to missing modalities at inference time. Furthermore, we employ a diffusion policy to learn the joint conditional probability distribution of physically-grounded trajectories and a intrinsic traversability heuristic. MAMMOTH utilizes this heuristic to prefer safer, smoother trajectories. We validate MAMMOTH through extensive real-world robot experiments in distinct off-road environments, including night-time operation. Our results demonstrate superior performance, with significant improvements in collision avoidance, terrain-aware planning and generalization to missing modalities. The code and dataset used for this work will be made publicly available.

Problem

• off-road navigation
• multi-modal perception
• sensor degradation
• illumination variation
• modality robustness

Innovation

• multi-modal fusion
• modality dropout
• diffusion policy
• off-road navigation
• traversability heuristic