NEMESIS: Noise-suppressed Efficient MAE with Enhanced Superpatch Integration Strategy

2026-04-02

AI Summary

This work addresses the challenges of high memory consumption and difficulty in modeling anisotropic structures in self-supervised learning from 3D CT images by proposing a masked autoencoder framework based on local 128³ supervoxels. The approach introduces three key innovations: a noise-augmented reconstruction pretraining task, an anatomical Transformer module featuring a face-axis dual-path masking mechanism, and NEMESIS Tokens for cross-scale contextual aggregation. The method substantially reduces computational cost—requiring only 31.0 GFLOPs per forward pass—while preserving fine-grained anatomical details. On the BTCV dataset, it achieves an average AUROC of 0.9633 using a frozen backbone with a linear classifier; remarkably, even with only 10% of the labels, it maintains strong performance at 0.9075 AUROC.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Segmentation
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Volumetric CT imaging is essential for clinical diagnosis, yet annotating 3D volumes is expensive and time-consuming, motivating self-supervised learning (SSL) from unlabeled data. However, applying SSL to 3D CT remains challenging due to the high memory cost of full-volume transformers and the anisotropic spatial structure of CT data, which is not well captured by conventional masking strategies. We propose NEMESIS, a masked autoencoder (MAE) framework that operates on local 128x128x128 superpatches, enabling memory-efficient training while preserving anatomical detail. NEMESIS introduces three key components: (i) noise-enhanced reconstruction as a pretext task, (ii) Masked Anatomical Transformer Blocks (MATB) that perform dual-masking through parallel plane-wise and axis-wise token removal, and (iii) NEMESIS Tokens (NT) for cross-scale context aggregation. On the BTCV multi-organ classification benchmark, NEMESIS with a frozen backbone and a linear classifier achieves a mean AUROC of 0.9633, surpassing fully fine-tuned SuPreM (0.9493) and VoCo (0.9387). Under a low-label regime with only 10% of available annotations, it retains an AUROC of 0.9075, demonstrating strong label efficiency. Furthermore, the superpatch-based design reduces computational cost to 31.0 GFLOPs per forward pass, compared to 985.8 GFLOPs for the full-volume baseline, providing a scalable and robust foundation for 3D medical imaging.

Problem

• self-supervised learning
• 3D CT imaging
• masked autoencoder
• anisotropic spatial structure
• memory efficiency

Innovation

• masked autoencoder
• superpatch
• noise-enhanced reconstruction
• anisotropic masking
• self-supervised learning