Repurposing RGB-based Foundation Model for Depth Estimation on Thermal Images Using Hierarchical Supervision

2026-08-11

AI Summary

This work addresses the challenge of thermal depth estimation for robotics under low-visibility conditions such as nighttime or adverse weather, where existing approaches struggle to effectively leverage the rich multi-level representations of RGB foundation models. To this end, we propose the RGB-HS framework, a dual-branch architecture that shares a common foundation model encoder between thermal and RGB modalities and enables knowledge transfer through cross-modal, multi-level token alignment. A novel token weighting mechanism, conditioned on RGB image quality, is introduced to refine the supervision signals during alignment. Experimental results demonstrate that our method significantly improves thermal depth estimation performance on standard benchmarks, effectively unlocking the representational potential of pre-trained RGB foundation models for thermal imaging tasks.

Abstract

Depth estimation from thermal images is highly valuable for robotic applications in adverse conditions, such as nighttime and rainy weather. Recent studies have sought to transfer knowledge from RGB-based foundation models to thermal modalities, yet the rich hierarchical representations these models encode remain underutilized. To address this limitation, we propose RGB-HS, a novel framework for thermal-image depth estimation that leverages hierarchical supervision from an RGB-based foundation model. Specifically, we first replace the baseline thermal encoder with a foundational model and introduce a parallel RGB branch that also employs a foundational model as an encoder of the same architecture, taking RGB images as input. The alignment is then performed across multiple levels between the tokens of the two encoders, allowing the thermal student branch to capture both structural precision and semantic abstraction from the RGB teacher branch. Furthermore, we introduce verification to refine the alignment process by weighting tokens from the RGB branch based on RGB image quality. Extensive experiments on the popular benchmark demonstrate that RGB-HS achieves competitive performance and more effectively exploits the representational capacity of RGB-based foundation models for depth estimation on thermal images.

Problem

• depth estimation
• thermal images
• foundation models
• hierarchical representation
• cross-modal transfer

Innovation

• hierarchical supervision
• foundation model
• thermal depth estimation
• cross-modal alignment
• RGB-to-thermal transfer