Mask-guided cross-image attention for zero-shot in-silico histopathologic image generation with a diffusion model

2024-07-16

arXiv.org

AI Summary

Zero-shot virtual generation of immunohistochemistry (IHC) images in computational pathology remains challenging, as existing appearance transfer models rely on foreground/background assumptions incompatible with the diagnostic relevance of *all* tissue regions in pathological images. Method: We propose a mask-driven, class-aware diffusion model that, for the first time, integrates segmentation-mask-guided cross-image attention and class-adaptive instance normalization (AdaIN)-based feature matching into a diffusion framework—enabling high-fidelity, tissue-level synthesis without paired data or target-domain priors. Contribution/Results: On epithelial cell segmentation, our method reduces manual annotation requirements by 75% while surpassing state-of-the-art baselines. Generated IHC images are validated by pathologists for clinical interpretability and effectively support both foundation model fine-tuning and supervised model training.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Creating in-silico data with generative AI promises a cost-effective alternative to staining, imaging, and annotating whole slide images in computational pathology. Diffusion models are the state-of-the-art solution for generating in-silico images, offering unparalleled fidelity and realism. Using appearance transfer diffusion models allows for zero-shot image generation, facilitating fast application and making model training unnecessary. However current appearance transfer diffusion models are designed for natural images, where the main task is to transfer the foreground object from an origin to a target domain, while the background is of insignificant importance. In computational pathology, specifically in oncology, it is however not straightforward to define which objects in an image should be classified as foreground and background, as all objects in an image may be of critical importance for the detailed understanding the tumor micro-environment. We contribute to the applicability of appearance transfer diffusion models to immunohistochemistry-stained images by modifying the appearance transfer guidance to alternate between class-specific AdaIN feature statistics matchings using existing segmentation masks. The performance of the proposed method is demonstrated on the downstream task of supervised epithelium segmentation, showing that the number of manual annotations required for model training can be reduced by 75%, outperforming the baseline approach. Additionally, we consulted with a certified pathologist to investigate future improvements. We anticipate this work to inspire the application of zero-shot diffusion models in computational pathology, providing an efficient method to generate in-silico images with unmatched fidelity and realism, which prove meaningful for downstream tasks, such as training existing deep learning models or finetuning foundation models.

Problem

• Tissue Imaging
• Cancer Research
• Digital Pathology

Innovation

• Improved Diffusion Model
• Zero-Shot Generation
• Pathological Image Analysis