🤖 AI Summary
This work addresses key challenges in thermal-infrared (T) and RGB dual-modal 3D scene reconstruction: cross-modal misalignment, single-modal overfitting, and low-fidelity real-time rendering. We propose the first 3D Gaussian Splatting framework tailored for thermal imaging. Methodologically, we introduce a thermal-aware 3D Gaussian representation; incorporate multi-modal regularization and thermal-physical priors into a smoothness constraint; and construct the first handheld real-world RGBT-Scenes dataset. Our contributions are threefold: (1) the first extension of 3D Gaussian Splatting to thermal imaging, enabling photorealistic thermal map rendering while simultaneously improving RGB reconstruction quality; (2) a 90% reduction in model size, enabling real-time dual-modal rendering; and (3) significant improvements in cross-modal geometric-radiometric consistency and generalization over existing single-modal approaches.
📝 Abstract
Thermography is especially valuable for the military and other users of surveillance cameras. Some recent methods based on Neural Radiance Fields (NeRF) are proposed to reconstruct the thermal scenes in 3D from a set of thermal and RGB images. However, unlike NeRF, 3D Gaussian splatting (3DGS) prevails due to its rapid training and real-time rendering. In this work, we propose ThermalGaussian, the first thermal 3DGS approach capable of rendering high-quality images in RGB and thermal modalities. We first calibrate the RGB camera and the thermal camera to ensure that both modalities are accurately aligned. Subsequently, we use the registered images to learn the multimodal 3D Gaussians. To prevent the overfitting of any single modality, we introduce several multimodal regularization constraints. We also develop smoothing constraints tailored to the physical characteristics of the thermal modality. Besides, we contribute a real-world dataset named RGBT-Scenes, captured by a hand-hold thermal-infrared camera, facilitating future research on thermal scene reconstruction. We conduct comprehensive experiments to show that ThermalGaussian achieves photorealistic rendering of thermal images and improves the rendering quality of RGB images. With the proposed multimodal regularization constraints, we also reduced the model's storage cost by 90%. The code and dataset will be released.