3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy

2026-06-22

AI Summary

This work addresses the limitations of existing self-supervised learning approaches for fluorescence microscopy, which often neglect the intrinsic three-dimensional cellular architecture by relying on two-dimensional projections. To overcome this, the authors propose a native 3D masked autoencoder and provide the first systematic validation demonstrating its superiority over 2D projection-based methods. The framework integrates the protein language model ESM2 to enable cross-modal alignment and incorporates channel cross-attention and frequency-domain regularization to enhance 3D spatial context modeling. Evaluated on protein–protein interaction prediction and protein subcellular localization tasks, the method achieves state-of-the-art performance with ROC-AUC of 0.865 and AUC_micro/F1_micro of 0.952/0.742, respectively, significantly advancing single-cell representation learning.

Abstract

Self-supervised learning in fluorescence microscopy often relies on 2D projections, despite the inherently three-dimensional nature of cells. We present a systematic comparison of 2D and 3D masked autoencoders (MAE-2D vs. MAE-3D) on volumetric microscopy data. Under matched architectures and training protocols, MAE-3D consistently outperforms 2D max-projection and slice-based variants on downstream single-cell tasks. We further align visual representations with a pretrained protein language model (ESM2) and show that cross-modal supervision yields larger gains for volumetric models. Channel cross-attention and frequency-domain regularization are critical for leveraging 3D spatial context. On a protein--protein interaction task, MAE-3D achieves a ROC--AUC of 0.865, outperforming prior methods by up to +0.025. For protein localization, our best 3D model attains state-of-the-art AUC$_{\text{micro}}$ (0.952) and F1$_{\text{micro}}$ (0.742), improving over previous approaches by +0.003 and +0.010 absolute, respectively. Overall, these results demonstrate the advantages of native 3D modeling and multimodal alignment for representation learning in single-cell microscopy.

Problem

• 3D masked autoencoders
• volumetric representation
• fluorescence microscopy
• multimodal alignment
• single-cell representation learning

Innovation

• 3D Masked Autoencoders
• Multimodal Alignment
• Volumetric Representation
• Cross-modal Supervision
• Frequency-domain Regularization