FGNet: Leveraging Feature-Guided Attention to Refine SAM2 for 3D EM Neuron Segmentation

2025-11-17

AI Summary

Segmentation of neuronal structures in electron microscopy (EM) images remains challenging due to complex morphology, low signal-to-noise ratio, and severe scarcity of annotated data, limiting both accuracy and generalizability. Method: We propose a 3D neuron segmentation framework leveraging knowledge transfer from SAM2. It incorporates a feature-guided attention module that exploits SAM2’s generic semantic priors to focus on ambiguous regions; a lightweight fine-grained encoder (FGE) to enhance local feature representation; and a dual-affinity decoder that jointly optimizes coarse-grained localization and fine-grained boundary delineation. Contribution/Results: Our method achieves state-of-the-art (SOTA) performance even with the SAM2 backbone frozen, and further surpasses existing approaches upon fine-tuning. This demonstrates the efficacy and robustness of adapting natural-image pretrained foundation models—via domain-adaptive strategies—to EM image segmentation, bridging the domain gap without requiring large-scale EM-specific pretraining.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Segmentation
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Accurate segmentation of neural structures in Electron Microscopy (EM) images is paramount for neuroscience. However, this task is challenged by intricate morphologies, low signal-to-noise ratios, and scarce annotations, limiting the accuracy and generalization of existing methods. To address these challenges, we seek to leverage the priors learned by visual foundation models on a vast amount of natural images to better tackle this task. Specifically, we propose a novel framework that can effectively transfer knowledge from Segment Anything 2 (SAM2), which is pre-trained on natural images, to the EM domain. We first use SAM2 to extract powerful, general-purpose features. To bridge the domain gap, we introduce a Feature-Guided Attention module that leverages semantic cues from SAM2 to guide a lightweight encoder, the Fine-Grained Encoder (FGE), in focusing on these challenging regions. Finally, a dual-affinity decoder generates both coarse and refined affinity maps. Experimental results demonstrate that our method achieves performance comparable to state-of-the-art (SOTA) approaches with the SAM2 weights frozen. Upon further fine-tuning on EM data, our method significantly outperforms existing SOTA methods. This study validates that transferring representations pre-trained on natural images, when combined with targeted domain-adaptive guidance, can effectively address the specific challenges in neuron segmentation.

Problem

• Addresses neuron segmentation challenges in EM images with complex morphologies
• Bridges domain gap between natural images and electron microscopy data
• Improves segmentation accuracy despite low signal-to-noise ratios

Innovation

• Feature-Guided Attention bridges domain gap
• Dual-affinity decoder generates coarse and refined maps
• Lightweight encoder focuses on challenging regions