Addressing Detail Bottlenecks in Latent Diffusion for RGB-to-SWIR Image Translation

2026-06-18

AI Summary

Existing latent diffusion models for RGB-to-SWIR image translation suffer from spatial detail degradation due to encoder compression and downsampling in the conditioning pathway, adversely affecting downstream perception performance. To address this, this work proposes two lightweight, backbone-agnostic enhancements: a Source-Conditioned Autoencoder (SCAE) and a Learnable Guidance Encoder (LGE). These modules inject source features via skip connections and replace naive downsampling with learnable encoding, significantly improving high-resolution detail preservation within both U-Net and DiT denoising frameworks. Experiments demonstrate that the proposed approach boosts detection mAP by up to 2× (3.4× for small objects) in driving scenarios, achieves state-of-the-art FID scores, and exhibits strong zero-shot generalization on the RASMD benchmark. Notably, the results reveal a weak correlation between FID and detection performance.

Abstract

Latent diffusion models (LDMs) enable efficient image-to-image translation but discard fine spatial details during compression, degrading downstream perception tasks. We identify two bottlenecks: the autoencoder, which loses spatial information, and the conditioning pathway, which further degrades the source signal through naive downsampling. We propose two lightweight, backbone-agnostic fixes: a Source-Conditioned Autoencoder (SCAE) that injects high-resolution source features into the decoder via skip connections, and a Learnable Guidance Encoder (LGE) that replaces naive downsampling with a learned conditioning signal. Evaluated on RGB-to-SWIR translation for driving scenes with two denoiser backbones (U-Net and DiT), our approach improves detection mAP by up to 2x over the latent diffusion baseline, with up to 3.4x gains on small objects (COCO-small, <32^2 px^2), while achieving state-of-the-art FID. We further show that FID and detection performance are poorly correlated, motivating multi-axis evaluation. Results generalise zero-shot to the public RASMD benchmark. We will publicly release test data with annotations, all checkpoints, and training code.

Problem

• Latent Diffusion
• RGB-to-SWIR Translation
• Detail Bottlenecks
• Image-to-Image Translation
• Spatial Detail Loss

Innovation

• Latent Diffusion Models
• RGB-to-SWIR Translation
• Source-Conditioned Autoencoder
• Learnable Guidance Encoder
• Detail Preservation