Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization

2026-03-19

AI Summary

This work addresses the high computational cost and poor cross-dataset generalization of existing methods for nuclei segmentation in histopathology images, which hinder their practical deployment. The authors propose a prompt-free, lightweight adaptation framework for the Segment Anything Model (SAM), wherein the original SAM encoder is frozen and only a 4.1M-parameter Low-Rank Adaptation (LoRA) module is fine-tuned. By integrating multi-level feature fusion and a residual decoder architecture, the method achieves both efficiency and high segmentation accuracy. Evaluated on three benchmarks—TNBC, MoNuSeg, and PanNuke—it attains state-of-the-art performance, marking the first successful realization of a low-overhead, highly generalizable, prompt-free SAM adaptation that balances effectiveness with practical utility.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Histopathology nuclei segmentation is crucial for quantitative tissue analysis and cancer diagnosis. Although existing segmentation methods have achieved strong performance, they are often computationally heavy and show limited generalization across datasets, which constrains their practical deployment. Recent SAM-based approaches have shown great potential in general and medical imaging, but typically rely on prompt guidance or complex decoders, making them less suitable for histopathology images with dense nuclei and heterogeneous appearances. We propose a prompt-free and lightweight SAM adaptation that leverages multi-level encoder features and residual decoding for accurate and efficient nuclei segmentation. The framework fine-tunes only LoRA modules within the frozen SAM encoder, requiring just 4.1M trainable parameters. Experiments on three benchmark datasets TNBC, MoNuSeg, and PanNuke demonstrate state-of-the-art performance and strong cross-dataset generalization, highlighting the effectiveness and practicality of the proposed framework for histopathology applications.

Problem

• histopathology
• nuclei segmentation
• cross-dataset generalization
• lightweight
• prompt-free

Innovation

• prompt-free
• lightweight SAM adaptation
• LoRA fine-tuning
• cross-dataset generalization
• nuclei segmentation