A scalar per patch from pre-trained ViTs enables fast moving navigation in the real world

2026-06-19

AI Summary

This work addresses the challenge of efficiently leveraging pretrained vision encoders to enhance real-world robotic navigation performance. To overcome limitations of existing approaches—particularly their overreliance on RGB inputs and inefficient feature utilization—the study proposes a framework integrating heterogeneous multi-teacher knowledge distillation, spatially structured feature bottlenecks, and policy fine-tuning guided by privileged information. Notably, the authors demonstrate that effective navigation can be achieved using only a single scalar per image patch. Evaluated across 24 kilometers of real-world trajectories encompassing 966 navigation tasks, the method significantly improves both efficiency and robustness. Furthermore, it uncovers interpretable visual features strongly correlated with environmental affordances, thereby identifying the visual elements most critical for robotic task success.

Abstract

Trained policies for real-world robotics rely on computer vision components, typically in the form of pre-trained visual encoders. These encoders are an essential component and it has been shown that their power does not emerge from training on robotics downstream losses alone. Pre-training with auxiliary losses in the form of computer-vision pre-text tasks is a defining factor and heavily conditions agent performance in robotics tasks. In this unprecedented large-scale study, we ran 966 navigation episodes of static point goal navigation in a real-world building for 24km and asked which components really matter for the computer vision aspects of robotics: we evaluate state-of-the art visual encoders in realistic conditions. We explore the usefulness of heterogeneous multi-teacher distillation leading to encoders with multiple different and complementary skills. We investigate how much information from these encoders is necessary for robotics by bottlenecking them in a principled and spatially useful way and we show that this leads to the emergence of interpretable features linked to affordances. We also argue that training policies on RGB data alone does not lead to an optimal usage of visual features and show this by finetuning policies pre-trained on privileged information. All in all, we paint a more complete picture of what aspects of computer vision are relevant for real-world navigation.

Problem

• visual encoders
• real-world navigation
• pre-training
• robotics
• computer vision

Innovation

• vision bottleneck
• heterogeneous distillation
• affordance-aware features
• real-world navigation
• pre-trained ViTs