RGB Pre-Training Enhanced Unobservable Feature Latent Diffusion Model for Spectral Reconstruction

2025-07-17

AI Summary

To address the challenge of estimating unobservable spectral features in RGB-to-hyperspectral image (HSI) reconstruction, this paper proposes a knowledge transfer framework leveraging a pre-trained RGB diffusion model. The core innovation lies in adapting a pre-trained latent diffusion model (LDM) for RGB images into an “invisible-spectral-feature diffusion model”: missing spectral dimensions are decoupled and compressed, enabling joint spectral-spatial modeling within a compact latent space. The method adopts a two-stage architecture—first, a Spectral Unobservable Feature Autoencoder (SpeUAE) extracts the 3D spectral manifold; then, a Spatial Autoencoder (SpaAE) and the adapted LDM jointly learn its spectral-spatial distribution. Evaluated on multiple benchmarks, the approach achieves state-of-the-art reconstruction accuracy and significantly improves downstream tasks such as relighting, demonstrating superior spectral fidelity and spatial-structural consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Spectral reconstruction (SR) is a crucial problem in image processing that requires reconstructing hyperspectral images (HSIs) from the corresponding RGB images. A key difficulty in SR is estimating the unobservable feature, which encapsulates significant spectral information not captured by RGB imaging sensors. The solution lies in effectively constructing the spectral-spatial joint distribution conditioned on the RGB image to complement the unobservable feature. Since HSIs share a similar spatial structure with the corresponding RGB images, it is rational to capitalize on the rich spatial knowledge in RGB pre-trained models for spectral-spatial joint distribution learning. To this end, we extend the RGB pre-trained latent diffusion model (RGB-LDM) to an unobservable feature LDM (ULDM) for SR. As the RGB-LDM and its corresponding spatial autoencoder (SpaAE) already excel in spatial knowledge, the ULDM can focus on modeling spectral structure. Moreover, separating the unobservable feature from the HSI reduces the redundant spectral information and empowers the ULDM to learn the joint distribution in a compact latent space. Specifically, we propose a two-stage pipeline consisting of spectral structure representation learning and spectral-spatial joint distribution learning to transform the RGB-LDM into the ULDM. In the first stage, a spectral unobservable feature autoencoder (SpeUAE) is trained to extract and compress the unobservable feature into a 3D manifold aligned with RGB space. In the second stage, the spectral and spatial structures are sequentially encoded by the SpeUAE and the SpaAE, respectively. The ULDM is then acquired to model the distribution of the coded unobservable feature with guidance from the corresponding RGB images. Experimental results on SR and downstream relighting tasks demonstrate that our proposed method achieves state-of-the-art performance.

Problem

• Reconstruct hyperspectral images from RGB images
• Estimate unobservable spectral features from RGB data
• Learn spectral-spatial joint distribution using RGB pre-trained models

Innovation

• Extends RGB-LDM to ULDM for spectral reconstruction
• Uses two-stage pipeline for spectral-spatial learning
• Separates unobservable feature to reduce redundancy