TopoMortar: A dataset to evaluate image segmentation methods focused on topology accuracy

2025-03-05

AI Summary

This paper addresses the challenge of evaluating topological accuracy in image segmentation. To this end, we introduce TopoMortar—the first controllable brick-wall benchmark dataset specifically designed for topology-aware evaluation—featuring controlled label noise, scalable training set sizes, and explicit in-distribution (ID) / out-of-distribution (OOD) test splits to disentangle contributions from topological priors versus data artifacts. We propose a topology-oriented evaluation paradigm, incorporating three semantic label types, dual-scale training sets, and an OOD test set to enable fine-grained attribution of topological performance. Experimental results demonstrate that clDice achieves the highest topological accuracy, while Skeleton Recall exhibits the strongest robustness to label noise. Moreover, self-distillation and data augmentation significantly enhance the topological performance of the CE+Dice loss, enabling it to surpass most dedicated topological losses. Finally, skeleton-based losses prove both computationally efficient and highly topologically robust.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

We present TopoMortar, a brick wall dataset that is the first dataset specifically designed to evaluate topology-focused image segmentation methods, such as topology loss functions. TopoMortar enables to investigate in two ways whether methods incorporate prior topological knowledge. First, by eliminating challenges seen in real-world data, such as small training set, noisy labels, and out-of-distribution test-set images, that, as we show, impact the effectiveness of topology losses. Second, by allowing to assess in the same dataset topology accuracy across dataset challenges, isolating dataset-related effects from the effect of incorporating prior topological knowledge. In these two experiments, it is deliberately difficult to improve topology accuracy without actually using topology information, thus, permitting to attribute an improvement in topology accuracy to the incorporation of prior topological knowledge. To this end, TopoMortar includes three types of labels (accurate, noisy, pseudo-labels), two fixed training sets (large and small), and in-distribution and out-of-distribution test-set images. We compared eight loss functions on TopoMortar, and we found that clDice achieved the most topologically accurate segmentations, Skeleton Recall loss performed best particularly with noisy labels, and the relative advantageousness of the other loss functions depended on the experimental setting. Additionally, we show that simple methods, such as data augmentation and self-distillation, can elevate Cross entropy Dice loss to surpass most topology loss functions, and that those simple methods can enhance topology loss functions as well. clDice and Skeleton Recall loss, both skeletonization-based loss functions, were also the fastest to train, making this type of loss function a promising research direction. TopoMortar and our code can be found at https://github.com/jmlipman/TopoMortar

Problem

• Evaluates topology-focused image segmentation methods.
• Assesses topology accuracy across dataset challenges.
• Compares effectiveness of various topology loss functions.

Innovation

• TopoMortar dataset evaluates topology-focused segmentation methods.
• Includes accurate, noisy, and pseudo-labels for diverse testing.
• clDice and Skeleton Recall loss achieve topologically accurate results.