SAM4EM: Efficient memory-based two stage prompt-free segment anything model adapter for complex 3D neuroscience electron microscopy stacks

2025-04-30

AI Summary

Automated 3D segmentation of fine subcellular structures—including mitochondria, astrocytic protrusions, and synapses—in electron microscopy (EM) neuroimaging data remains challenging due to their small size, morphological complexity, and low contrast. Method: We propose a prompt-free two-stage adaptation framework comprising: (1) a novel prompt-free adapter combined with LoRA-driven dual-stage fine-tuning; (2) a 3D memory attention mechanism that explicitly models cross-layer spatial consistency; and (3) a self-generated prompt embedding module coupled with a two-stage mask decoding architecture. Contribution/Results: We introduce the first dedicated 3D EM benchmark dataset for astrocytic protrusions and synapses. Our method achieves significant improvements over state-of-the-art methods across multiple challenging structures—particularly in segmenting postsynaptic densities and astrocytic protrusions. All code and trained models are fully open-sourced.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We present SAM4EM, a novel approach for 3D segmentation of complex neural structures in electron microscopy (EM) data by leveraging the Segment Anything Model (SAM) alongside advanced fine-tuning strategies. Our contributions include the development of a prompt-free adapter for SAM using two stage mask decoding to automatically generate prompt embeddings, a dual-stage fine-tuning method based on Low-Rank Adaptation (LoRA) for enhancing segmentation with limited annotated data, and a 3D memory attention mechanism to ensure segmentation consistency across 3D stacks. We further release a unique benchmark dataset for the segmentation of astrocytic processes and synapses. We evaluated our method on challenging neuroscience segmentation benchmarks, specifically targeting mitochondria, glia, and synapses, with significant accuracy improvements over state-of-the-art (SOTA) methods, including recent SAM-based adapters developed for the medical domain and other vision transformer-based approaches. Experimental results indicate that our approach outperforms existing solutions in the segmentation of complex processes like glia and post-synaptic densities. Our code and models are available at https://github.com/Uzshah/SAM4EM.

Problem

• Segment complex 3D neural structures in EM data
• Improve segmentation accuracy with limited annotated data
• Ensure 3D consistency in neuroscience electron microscopy stacks

Innovation

• Prompt-free adapter with two-stage mask decoding
• Dual-stage fine-tuning using Low-Rank Adaptation
• 3D memory attention for cross-stack consistency