SAM3-UNet: Simplified Adaptation of Segment Anything Model 3

2025-12-01

AI Summary

To address the high deployment cost and excessive GPU memory consumption of Segment Anything Model 3 (SAM3) in downstream tasks, this work proposes a lightweight adaptation framework. The method retains SAM3’s frozen image encoder and introduces parameter-efficient adapter modules, coupled with a streamlined U-Net–style decoder, significantly reducing computational and memory overhead. Evaluated on mirror detection and salient object detection, our framework achieves state-of-the-art performance—outperforming baselines such as SAM2-UNet—while requiring less than 6 GB GPU memory for training (batch size = 12), representing over a 60% reduction compared to full SAM3 fine-tuning. Our core contribution is the first efficient multi-task transfer adaptation of SAM3, achieving an optimal trade-off among accuracy, inference speed, and resource efficiency—thereby enabling practical deployment on edge devices.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In this paper, we introduce SAM3-UNet, a simplified variant of Segment Anything Model 3 (SAM3), designed to adapt SAM3 for downstream tasks at a low cost. Our SAM3-UNet consists of three components: a SAM3 image encoder, a simple adapter for parameter-efficient fine-tuning, and a lightweight U-Net-style decoder. Preliminary experiments on multiple tasks, such as mirror detection and salient object detection, demonstrate that the proposed SAM3-UNet outperforms the prior SAM2-UNet and other state-of-the-art methods, while requiring less than 6 GB of GPU memory during training with a batch size of 12. The code is publicly available at https://github.com/WZH0120/SAM3-UNet.

Problem

• Adapts SAM3 for low-cost downstream segmentation tasks
• Uses parameter-efficient fine-tuning with a lightweight decoder
• Achieves superior performance with reduced GPU memory usage

Innovation

• Simplified SAM3 variant for low-cost adaptation
• Parameter-efficient fine-tuning with simple adapter
• Lightweight U-Net decoder for efficient training