Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesis

2026-07-02

AI Summary

This work addresses the challenge of synthesizing novel-view images in non-RGB modalities—such as infrared, polarization, and multispectral—without requiring expensive specialized sensors or per-scene multimodal data acquisition. The authors propose SPoILeR, a method that leverages multimodal pretraining to model cross-modality relationships and integrates implicit neural representations with an RGB-supervised fine-tuning strategy. Remarkably, SPoILeR enables high-quality, geometrically and photometrically consistent synthesis of target modality images from only a single RGB image of a scene, without any input from the target modality itself. Experimental results demonstrate accurate and consistent performance across infrared, polarization, and multispectral rendering tasks, significantly reducing reliance on multimodal data collection and exhibiting strong generalization capabilities.

Abstract

Neural rendering techniques allow for accurate reconstruction of the geometry and color appearance of 3D scenes. Some methods have extended their use to additional imaging modalities, such as multispectral, infrared, or polarimetric data. However, all of these approaches require expensive sensors and calibrated setups to capture new multimodal frames for each new scene. We propose Spectral and Polarimetric Implicit Learned Representation (SPoILeR), a novel method to obtain multi-view consistent renderings of unconventional modalities for scenes where either only RGB frames or very few of the additional modalities are available. Thanks to a multimodal pre-training phase, the model learns the mutual correlation between different modalities. This step allows predicting accurate renderings of unconventional modalities during a fine-tuning phase supervised only by RGB images. Experimental results show that the approach can accurately render infrared, polarimetric, and multispectral frames for scenes where no input sample captured by these types of sensors is provided.

Problem

• novel view synthesis
• multimodal rendering
• spectral imaging
• polarimetric imaging
• neural rendering

Innovation

• multimodal pre-training
• novel view synthesis
• spectral rendering
• polarimetric imaging
• neural rendering