Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation

2026-08-25

AI Summary

为解决手术器械分割中模型适应性差的问题，本文提出了HPMA框架，通过构建多尺度视觉原型记忆库并引入尺度匹配耦合机制来提高SAM模型的性能。

Abstract

Surgical instrument segmentation (SIS) is fundamental for computer-assisted surgery, where reliable instrument masks enable precise scene understanding and clinical assistance. Recently, adapting foundation models like the Segment Anything Model (SAM) to the surgical domain via prompt-learning has shown encouraging results. However, the performance of these adapted models under challenging surgical conditions is constrained by suboptimal adaptation mechanisms. Specifically, optimizing prompts or prototypes purely via downstream segmentation loss tends to cause them to degenerate into task-specific parameters rather than serving as persistent, stable category memory, thereby degrading their robustness against complex intraoperative variations. Moreover, routing multi-scale visual cues through a single prompt pathway creates a bottleneck that hinders effective scale-matched coupling. To address these limitations, we propose HPMA, a Hierarchical Prototype-Memory Adaptation framework for SAM. Specifically, HPMA constructs a frozen, multi-scale visual prototype memory bank from annotated surgical scenes and integrates it into SAM's feature space using lightweight adapters to preserve stable category evidence. To maximize the utility of multi-scale cues, we introduce a scale-matched coupling mechanism where global prototypes calibrate class-level prompt features, structural prototypes guide decoder object queries, and local prototypes align high-resolution feature maps through a local alignment objective. Extensive experiments on the public EndoVis2017 and EndoVis2018 datasets demonstrate that our approach achieves state-of-the-art performance, outperforming existing foundation model adaptation methods.

Problem

• Surgical Instrument Segmentation
• Adaptation Mechanisms
• Multi-scale Visual Cues
• Robustness
• Prototype Degeneration

Innovation

• Hierarchical Prototype-Memory Adaptation
• multi-scale visual prototype memory bank
• scale-matched coupling mechanism
• lightweight adapters