SAM$^{*}$: Task-Adaptive SAM with Physics-Guided Rewards

2025-09-08

AI Summary

Microscopic image segmentation faces challenges of complex hyperparameter tuning and poor real-time performance when adapting foundation models. This paper proposes a physics-guided, reward-driven adaptive optimization framework that dynamically fine-tunes the Segment Anything Model (SAM) via a differentiable reward function. Physical priors—including particle size distribution and geometric shape constraints—are explicitly encoded into the optimization objective, enabling task-adaptive segmentation without manual parameter adjustment. The method significantly improves segmentation accuracy and robustness for cellular structures, material interfaces, and nanoscale features, achieving high-throughput, low-latency streaming inference across diverse microscopic imaging modalities. Its core innovation lies in the first formulation of explicit physical constraints as differentiable rewards, endowing foundation models with interpretable and generalizable online optimization capability. This establishes an efficient, lightweight segmentation paradigm for computational microscopy.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Optimization

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Image segmentation is a critical task in microscopy, essential for accurately analyzing and interpreting complex visual data. This task can be performed using custom models trained on domain-specific datasets, transfer learning from pre-trained models, or foundational models that offer broad applicability. However, foundational models often present a considerable number of non-transparent tuning parameters that require extensive manual optimization, limiting their usability for real-time streaming data analysis. Here, we introduce a reward function-based optimization to fine-tune foundational models and illustrate this approach for SAM (Segment Anything Model) framework by Meta. The reward functions can be constructed to represent the physics of the imaged system, including particle size distributions, geometries, and other criteria. By integrating a reward-driven optimization framework, we enhance SAM's adaptability and performance, leading to an optimized variant, SAM$^{*}$, that better aligns with the requirements of diverse segmentation tasks and particularly allows for real-time streaming data segmentation. We demonstrate the effectiveness of this approach in microscopy imaging, where precise segmentation is crucial for analyzing cellular structures, material interfaces, and nanoscale features.

Problem

• Optimizing foundational models' non-transparent tuning parameters
• Enabling real-time segmentation for streaming microscopy data
• Improving segmentation accuracy for diverse physical criteria

Innovation

• Physics-guided reward functions optimize SAM
• Real-time streaming data segmentation enabled
• Fine-tuning with particle size and geometry criteria