Towards Integrating Multi-Spectral Imaging with Gaussian Splatting

2025-08-31

AI Summary

This paper addresses reconstruction degradation in multispectral multi-view 3D reconstruction caused by geometric inconsistency across spectral bands. We propose a novel framework integrating multispectral imaging with 3D Gaussian Splatting (3DGS). Our key contributions are: (1) joint embedding of red, green, red-edge, and near-infrared band data into spherical harmonic color coefficients to construct a compact, spectrally consistent reflectance representation; and (2) a multi-stage joint optimization and cross-spectral collaborative training strategy to co-model geometric and spectral features. Experiments on our newly collected multispectral dataset demonstrate significant improvements in geometric fidelity and spectral consistency for novel view synthesis—achieving average PSNR gains of 2.1 dB and SSIM improvements of 0.042. Qualitative results confirm robust performance in complex scenes involving vegetation and heterogeneous materials.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We present a study of how to integrate color (RGB) and multi-spectral imagery (red, green, red-edge, and near-infrared) into the 3D Gaussian Splatting (3DGS) framework, a state-of-the-art explicit radiance-field-based method for fast and high-fidelity 3D reconstruction from multi-view images. While 3DGS excels on RGB data, naive per-band optimization of additional spectra yields poor reconstructions due to inconsistently appearing geometry in the spectral domain. This problem is prominent, even though the actual geometry is the same, regardless of spectral modality. To investigate this, we evaluate three strategies: 1) Separate per-band reconstruction with no shared structure. 2) Splitting optimization, in which we first optimize RGB geometry, copy it, and then fit each new band to the model by optimizing both geometry and band representation. 3) Joint, in which the modalities are jointly optimized, optionally with an initial RGB-only phase. We showcase through quantitative metrics and qualitative novel-view renderings on multi-spectral datasets the effectiveness of our dedicated optimized Joint strategy, increasing overall spectral reconstruction as well as enhancing RGB results through spectral cross-talk. We therefore suggest integrating multi-spectral data directly into the spherical harmonics color components to compactly model each Gaussian's multi-spectral reflectance. Moreover, our analysis reveals several key trade-offs in when and how to introduce spectral bands during optimization, offering practical insights for robust multi-modal 3DGS reconstruction.

Problem

• Integrating RGB and multi-spectral imagery into 3D Gaussian Splatting
• Addressing inconsistent geometry reconstruction across spectral bands
• Optimizing joint multi-spectral representation within 3DGS framework

Innovation

• Integrating RGB and multi-spectral imagery into 3DGS
• Joint optimization strategy with shared geometry
• Modeling multi-spectral reflectance via spherical harmonics