Multi-Spectral Gaussian Splatting with Neural Color Representation

2025-06-03

AI Summary

This work addresses novel view synthesis under multi-spectral heterogeneous cameras (e.g., thermal infrared, near-infrared) without cross-modal calibration. We propose a multi-spectral 3D Gaussian splatting reconstruction framework. Its core innovation lies in introducing learnable, per-Gaussian unified feature embeddings, jointly optimized with neural color representation and a lightweight MLP-based spectral decoder to explicitly model spectral–spatial correlations—replacing conventional channel-wise spherical harmonic optimization. The framework requires no cross-modal camera calibration and supports flexible integration of arbitrary spectral bands. Evaluated on agricultural remote sensing tasks, it achieves significant improvements in vegetation index rendering accuracy (e.g., NDVI), outperforms state-of-the-art methods in PSNR and SSIM across all individual spectral bands, and maintains real-time rendering efficiency.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting (3DGS) framework that is able to generate multi-view consistent novel views from images of multiple, independent cameras with different spectral domains. In contrast to previous approaches, our method does not require cross-modal camera calibration and is versatile enough to model a variety of different spectra, including thermal and near-infra red, without any algorithmic changes. Unlike existing 3DGS-based frameworks that treat each modality separately (by optimizing per-channel spherical harmonics) and therefore fail to exploit the underlying spectral and spatial correlations, our method leverages a novel neural color representation that encodes multi-spectral information into a learned, compact, per-splat feature embedding. A shallow multi-layer perceptron (MLP) then decodes this embedding to obtain spectral color values, enabling joint learning of all bands within a unified representation. Our experiments show that this simple yet effective strategy is able to improve multi-spectral rendering quality, while also leading to improved per-spectra rendering quality over state-of-the-art methods. We demonstrate the effectiveness of this new technique in agricultural applications to render vegetation indices, such as normalized difference vegetation index (NDVI).

Problem

• Enables multi-spectral 3D view synthesis without cross-modal camera calibration
• Unifies spectral bands via neural feature embedding for joint learning
• Improves rendering quality for diverse spectra including thermal and infrared

Innovation

• Multi-spectral 3D Gaussian Splatting framework
• Neural color representation for spectral embedding
• Shallow MLP decodes multi-spectral features