Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesis

2026-03-22

AI Summary

This work addresses the limitations of existing counterfactual image generation methods, which struggle to achieve spatially localized structural modifications and often introduce global artifacts, typically relying on manual masks or supporting only global interventions. To overcome these challenges, the authors propose a region-level counterfactual generation framework that decomposes anatomical structures into independent regional segments and leverages segmentation-derived regional measurements for supervised fine-tuning. This approach enables precise, mask-free regional interventions for the first time. Experiments on coronary computed tomography angiography (CTA) data demonstrate that the method generates anatomically consistent and realistic images with region-specific lesion simulations, significantly improving the spatial precision of disease progression modeling.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Structured Learning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Counterfactual image generation enables controlled data augmentation, bias mitigation, and disease modeling. However, existing methods guided by external classifiers or regressors are limited to subject-level factors (e.g., age) and fail to produce localized structural changes, often resulting in global artifacts. Pixel-level guidance using segmentation masks has been explored, but requires user-defined counterfactual masks, which are tedious and impractical. Segmentor-guided Counterfactual Fine-Tuning (Seg-CFT) addressed this by using segmentation-derived measurements to supervise structure-specific variables, yet it remains restricted to global interventions. We propose Positional Seg-CFT, which subdivides each structure into regional segments and derives independent measurements per region, enabling spatially localized and anatomically coherent counterfactuals. Experiments on coronary CT angiography show that Pos-Seg-CFT generates realistic, region-specific modifications, providing finer spatial control for modeling disease progression.

Problem

• counterfactual image generation
• spatially localized synthesis
• image segmentation
• anatomical coherence
• disease modeling

Innovation

• counterfactual generation
• spatially localized synthesis
• segmentation-guided fine-tuning
• anatomical coherence
• medical image augmentation