Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth

2026-03-01

AI Summary

This work addresses the challenge of multispectral demosaicing—reconstructing full-resolution spectral images from a single snapshot of mosaic measurements—where conventional methods often produce blurry results and supervised learning approaches rely on costly ground-truth data. To overcome these limitations, we propose the PEFD framework, which, for the first time, enables effective transfer of pretrained foundation models to multispectral demosaicing without requiring ground-truth supervision. PEFD integrates camera projection geometry modeling with viewpoint-equivariant fine-tuning and leverages group-theoretic structures to recover null-space information, achieving high-quality unsupervised reconstruction. Experiments on neurosurgical and autonomous driving datasets demonstrate that PEFD significantly outperforms existing unsupervised methods, yielding sharp details (e.g., blood vessels) and high spectral fidelity, with performance approaching that of supervised approaches.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multispectral demosaicing is crucial to reconstruct full-resolution spectral images from snapshot mosaiced measurements, enabling real-time imaging from neurosurgery to autonomous driving. Classical methods are blurry, while supervised learning requires costly ground truth (GT) obtained from slow line-scanning systems. We propose Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), a framework that learns multispectral demosaicing from mosaiced measurements alone. PEFD a) exploits the projective geometry of camera-based imaging systems to leverage a richer group structure than previous demosaicing methods to recover more null-space information, and b) learns efficiently without GT by adapting pretrained foundation models designed for 1-3 channel imaging. On intraoperative and automotive datasets, PEFD recovers fine details such as blood vessels and preserves spectral fidelity, substantially outperforming recent approaches, nearing supervised performance.

Problem

• multispectral demosaicing
• ground truth
• snapshot imaging
• spectral reconstruction

Innovation

• perspective-equivariance
• multispectral demosaicing
• self-supervised learning
• foundation model adaptation
• projective geometry