CF-Seg: Counterfactuals meet Segmentation

2025-06-19

AI Summary

Disease-induced tissue deformation, ambiguous boundaries, and structural occlusion in medical images severely compromise anatomical segmentation accuracy, thereby undermining quantitative assessment and clinical decision-making. To address this, we introduce counterfactual reasoning into medical image segmentation for the first time, proposing a decoupled, model-agnostic enhancement framework. Leveraging generative adversarial networks jointly optimized with anatomical prior constraints, our method synthesizes “disease-free” counterfactual images for the same patient—serving as robust inputs to off-the-shelf segmentation models without architectural modification. Crucially, this approach mitigates disease-related confounding effects at inference time without incurring additional computational overhead. Evaluated on two real-world chest X-ray datasets, it achieves an average Dice coefficient improvement of 4.2%, substantially enhancing segmentation robustness and downstream clinical reliability.

Technology Category

• Computer Vision: Segmentation
• Reasoning under Uncertainty: Causality
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Segmenting anatomical structures in medical images plays an important role in the quantitative assessment of various diseases. However, accurate segmentation becomes significantly more challenging in the presence of disease. Disease patterns can alter the appearance of surrounding healthy tissues, introduce ambiguous boundaries, or even obscure critical anatomical structures. As such, segmentation models trained on real-world datasets may struggle to provide good anatomical segmentation, leading to potential misdiagnosis. In this paper, we generate counterfactual (CF) images to simulate how the same anatomy would appear in the absence of disease without altering the underlying structure. We then use these CF images to segment structures of interest, without requiring any changes to the underlying segmentation model. Our experiments on two real-world clinical chest X-ray datasets show that the use of counterfactual images improves anatomical segmentation, thereby aiding downstream clinical decision-making.

Problem

• Improving segmentation of diseased medical images
• Generating counterfactual images for healthy anatomy
• Enhancing clinical decision-making via better segmentation

Innovation

• Generates counterfactual images for disease-free simulation
• Uses CF images to improve anatomical segmentation
• Requires no changes to existing segmentation models