Transformation Behavior of Images in Latent Space

2026-06-23

AI Summary

This study systematically investigates the impact of classical image transformations on the latent representations of histopathology image encoders, evaluating their degree of invariance to label-irrelevant augmentations. By measuring embedding-space distances among original images, standardly augmented variants, and randomly unrelated images, we conduct a comparative analysis using both general-purpose and pathology-specific encoders—namely those from Lunit, Bioptimus, and Meta—on colorectal H&E-stained whole-slide images and TCGA datasets. Our work quantitatively reveals, for the first time, that current encoders exhibit only partial invariance to common augmentation strategies, with notable differences between general and domain-specific models. Crucially, we find that post-transformation embeddings remain significantly closer to their originals than to random samples, thereby elucidating a key mechanism through which data augmentation enhances model performance.

Abstract

Training of neural networks for histopathology classification tasks typically relies on data encoding into latent space, which reduces complexity and improves performance. There are several encoder networks available, either pretrained on general image datasets such as ImageNET, or specifically on histopathological images. Training of encoder networks should be adapted to downstream tasks, allowing encoding of biologic/diagnostic content while rendering networks invariant to label-irrelevant transformations. This paper investigates the effect of classical image transformation on the latent space, using networks provided by Lunit Inc. and Bioptimus, both focusing on pathological images, and by Meta Research Team. We assess variance of embeddings resulting from standard data transformations by comparing original and transformed image embeddings and by contrasting them with random, unrelated embeddings, using image tiles from hematoxylin/eosin-stained sections available in a colorectal tissue dataset and the publicly accessible TCGA dataset. Our findings show that embeddings of original and transformed images are closer to each other than to random embeddings, indicating robustness to transformations. However, they are not fully invariant, revealing that the encoder networks do not completely neutralize transformation effects in latent space, explaining why transformation-mediated augmentation of datasets can improve performance. Significant differences were observed between general and histopathology-specific encoder networks.

Problem

• latent space
• image transformation
• histopathology
• encoder networks
• transformation invariance

Innovation

• latent space
• image transformation
• histopathology
• embedding invariance
• encoder robustness