Segmentor-Guided Counterfactual Fine-Tuning for Image Synthesis

2025-09-29

AI Summary

Existing counterfactual image generation methods struggle to achieve anatomically precise interventions—such as modulating left lung area in chest X-rays—often inducing global distortions and relying on scarce, pixel-level segmentation annotations. To address this, we propose a fully unsupervised, anatomy-aware counterfactual generation framework. Our approach is the first to embed a learnable, differentiable semantic segmentation module directly into the latent space of a generative model, enabling scalar variables to directly control anatomical attributes. Fine-grained latent editing, guided by the embedded segmentation module, ensures local anatomical consistency while preserving global image fidelity. Evaluated on chest X-ray data, our method successfully generates photorealistic counterfactual images exhibiting coronary artery disease hallmarks. It significantly enhances controllability and practicality for medical image augmentation and pathological modeling—without requiring any manual segmentation labels.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Counterfactual image generation is a powerful tool for augmenting training data, de-biasing datasets, and modeling disease. Current approaches rely on external classifiers or regressors to increase the effectiveness of subject-level interventions (e.g., changing the patient's age). For structure-specific interventions (e.g., changing the area of the left lung in a chest radiograph), we show that this is insufficient, and can result in undesirable global effects across the image domain. Previous work used pixel-level label maps as guidance, requiring a user to provide hypothetical segmentations which are tedious and difficult to obtain. We propose Segmentor-guided Counterfactual Fine-Tuning (Seg-CFT), which preserves the simplicity of intervening on scalar-valued, structure-specific variables while producing locally coherent and effective counterfactuals. We demonstrate the capability of generating realistic chest radiographs, and we show promising results for modeling coronary artery disease. Code: https://github.com/biomedia-mira/seg-cft.

Problem

• Generating structure-specific counterfactual images without global side effects
• Eliminating reliance on tedious manual segmentation for image guidance
• Producing locally coherent counterfactuals using scalar-valued structural variables

Innovation

• Segmentor-guided fine-tuning for counterfactual image synthesis
• Uses structure-specific variables for localized interventions
• Generates locally coherent counterfactuals without manual segmentations