SC-Diff: Semantically Calibrated Diffusion for Visible-to-Infrared Image Translation

2026-08-09

AI Summary

Visible-to-infrared image translation is often hindered by semantic layout distortions that degrade downstream task performance. To address this issue, this work proposes a semantics-calibrated latent diffusion framework that leverages a pretrained Segment Anything Model (SAM) to extract semantic masks, which are then converted into token-level labels. The method introduces a novel semantics-guided self-attention calibration (SGSC) mechanism within the denoising network to simultaneously enable conditional guidance and suppress cross-category interference. This approach effectively preserves object position, shape, and semantic consistency in the generated infrared images, significantly enhancing perceptual quality and yielding more reliable synthetic training data for infrared object detection.

Abstract

Visible-to-infrared image translation provides a practical way to expand infrared training data using abundant visible images. Diffusion models are promising for this task because of their strong generative performance. However, existing diffusion-based methods typically use semantic priors only as external conditions, without explicitly regulating token interactions within the denoising network. Consequently, they struggle to preserve object locations, shapes, and semantic layouts required for reliable annotation reuse. We propose SC-Diff, a semantically calibrated latent diffusion framework that uses semantic priors for both conditional guidance and internal self-attention calibration. A pretrained SAM3 model with predefined text prompts first extracts category-specific semantic masks from visible images. These masks are merged into a semantic map and fused with the visible image as the input condition. The same map is converted into token-level semantic labels to calibrate self-attention in the denoising network. Based on these labels, we introduce Semantic-Guided Self-Attention Calibration (SGSC), which adaptively applies positive biases to query-key pairs of the same category. The query-wise calibration strength depends on the dispersion of attention across semantic categories and the attention assigned to the query's own category. The original attention scores further modulate the bias, giving greater calibration to same-category keys with stronger responses. This soft calibration reduces cross-category interference while retaining global contextual interactions, thereby improving semantic consistency in generated infrared images. Extensive experiments show that SC-Diff improves perceptual quality and produces more effective synthetic training data for downstream infrared object detection.

Problem

• visible-to-infrared image translation
• diffusion models
• semantic consistency
• object detection
• image generation

Innovation

• Semantic Calibration
• Diffusion Model
• Visible-to-Infrared Translation
• Self-Attention Mechanism
• Synthetic Training Data