SegDem: Segmentation helps Demosaicing

2026-08-08

AI Summary

This work addresses the limitation of existing image demosaicking methods, which predominantly rely on local textures and low-level statistics while neglecting high-level scene structure. To overcome this, we propose SegDem, a novel framework that introduces instance segmentation into the demosaicking task for the first time. SegDem leverages a pre-trained instance segmentation model to learn region- and boundary-aware representations and transfers its decoder to RAW image reconstruction. Crucially, both the segmentation and demosaicking branches share a frozen DINOv2 feature space, ensuring structural consistency. The approach is compatible with diverse backbone architectures—including convolutional networks, Transformers, and state space models—and supports various color filter array layouts such as Bayer and Quad-Bayer. Extensive experiments demonstrate that SegDem consistently outperforms current state-of-the-art methods across synthetic, external, and challenging real-world datasets.

Abstract

Image demosaicing reconstructs a full-color image from incomplete color measurements produced by a sensor covered with a color filter array (CFA). Most existing methods formulate demosaicing as pixel-level reconstruction and mainly rely on local textures, cross-channel correlations, and low-level image statistics. Our core insight is that reconstruction and visual understanding can be viewed as complementary views of shared scene structure: both are grounded in the same underlying physical world, and therefore the structural and physical information inferred from an image should remain consistent across the two tasks. We instantiate this idea with instance segmentation and propose \emph{SegDem}, a cross-task decoder representation transfer framework for demosaicing. SegDem first learns region- and boundary-aware representations through instance-aware structural pretraining and then transfers the decoder to RAW-conditioned reconstruction. Segmentation- and demosaicing-conditioned features are further anchored to a shared frozen DINOv2 representation space to preserve structural organization across tasks. We instantiate SegDem with convolutional, Transformer-based, and state-space backbones for unified Single- and Quad-Bayer demosaicing. Extensive experiments on synthetic, external, and challenging datasets demonstrate consistent improvements across different architectures and CFA layouts.

Problem

• image demosaicing
• color filter array
• structural consistency
• visual understanding
• full-color image reconstruction

Innovation

• demosaicing
• instance segmentation
• cross-task representation transfer
• DINOv2
• structural consistency