Uncertainty-Aware ControlNet: Bridging Domain Gaps with Synthetic Image Generation

2025-10-13

AI Summary

ControlNet often replicates the source-domain distribution in cross-domain image synthesis, struggling to adapt to unlabeled target domains (e.g., low-quality Home-OCT OCT images). To address this, we propose Uncertainty-ControlNet—the first method to embed uncertainty modeling directly into ControlNet’s control mechanism. It jointly leverages semantic guidance from labeled source data and uncertainty-driven guidance from unlabeled target-domain inputs, enabling arbitrary-domain image generation without explicit style alignment. Our model integrates a diffusion-based architecture, dual-control inputs (semantic + uncertainty), and automatic synthetic annotation of generated images, supporting unsupervised synthesis of labeled data for the target domain. Evaluated on retinal OCT and traffic-scene cross-domain segmentation tasks, Uncertainty-ControlNet achieves effective domain adaptation using only synthetically labeled target-domain data—outperforming style-transfer-based and other baseline methods by significant margins in segmentation accuracy.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Diffusion Models for Vision
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Generative Models are a valuable tool for the controlled creation of high-quality image data. Controlled diffusion models like the ControlNet have allowed the creation of labeled distributions. Such synthetic datasets can augment the original training distribution when discriminative models, like semantic segmentation, are trained. However, this augmentation effect is limited since ControlNets tend to reproduce the original training distribution. This work introduces a method to utilize data from unlabeled domains to train ControlNets by introducing the concept of uncertainty into the control mechanism. The uncertainty indicates that a given image was not part of the training distribution of a downstream task, e.g., segmentation. Thus, two types of control are engaged in the final network: an uncertainty control from an unlabeled dataset and a semantic control from the labeled dataset. The resulting ControlNet allows us to create annotated data with high uncertainty from the target domain, i.e., synthetic data from the unlabeled distribution with labels. In our scenario, we consider retinal OCTs, where typically high-quality Spectralis images are available with given ground truth segmentations, enabling the training of segmentation networks. The recent development in Home-OCT devices, however, yields retinal OCTs with lower quality and a large domain shift, such that out-of-the-pocket segmentation networks cannot be applied for this type of data. Synthesizing annotated images from the Home-OCT domain using the proposed approach closes this gap and leads to significantly improved segmentation results without adding any further supervision. The advantage of uncertainty-guidance becomes obvious when compared to style transfer: it enables arbitrary domain shifts without any strict learning of an image style. This is also demonstrated in a traffic scene experiment.

Problem

• Bridging domain gaps in image data using synthetic generation
• Improving segmentation on unlabeled domains with uncertainty control
• Generating annotated synthetic data for domain adaptation tasks

Innovation

• Introducing uncertainty control into ControlNet mechanism
• Generating synthetic data from unlabeled target domains
• Enabling domain adaptation without additional supervision requirements