MSGS: Multispectral 3D Gaussian Splatting

2026-04-14

AI Summary

Traditional RGB-only novel view synthesis methods struggle to accurately model spectral characteristics, particularly when dealing with complex materials. This work proposes the first 3D Gaussian Splatting framework capable of multispectral rendering by introducing a spherical harmonics–based spectral radiance representation for each Gaussian and designing a differentiable, pixel-wise transformation from spectral to RGB space. The approach jointly optimizes the model using a dual supervision loss combining both RGB and multispectral signals, achieving spectral consistency while preserving real-time rendering efficiency and model compactness. Evaluated on both public and newly captured real-world datasets, the method significantly outperforms existing RGB-based baselines, demonstrating superior image quality and spectral fidelity—especially in scenes involving translucent and anisotropic materials.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present a multispectral extension to 3D Gaussian Splatting (3DGS) for wavelength-aware view synthesis. Each Gaussian is augmented with spectral radiance, represented via per-band spherical harmonics, and optimized under a dual-loss supervision scheme combining RGB and multispectral signals. To improve rendering fidelity, we perform spectral-to-RGB conversion at the pixel level, allowing richer spectral cues to be retained during optimization. Our method is evaluated on both public and self-captured real-world datasets, demonstrating consistent improvements over the RGB-only 3DGS baseline in terms of image quality and spectral consistency. Notably, it excels in challenging scenes involving translucent materials and anisotropic reflections. The proposed approach maintains the compactness and real-time efficiency of 3DGS while laying the foundation for future integration with physically based shading models.

Problem

• multispectral
• 3D Gaussian Splatting
• view synthesis
• spectral consistency
• wavelength-aware rendering

Innovation

• Multispectral
• 3D Gaussian Splatting
• Spectral Radiance
• Spherical Harmonics
• View Synthesis