Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning

2024-08-15

arXiv.org

AI Summary

Existing SAM2 models struggle with real-time surgical video segmentation due to excessive computational overhead and high GPU memory consumption, especially under high-resolution and long-duration scenarios. Method: We propose an Efficient Frame Pruning (EFP) mechanism that dynamically selects key frames via adaptive frame importance scoring, jointly reducing computation and memory footprint. We further integrate temporal memory management with lightweight fine-tuning—conducted exclusively at low resolution—to preserve state-of-the-art accuracy. Contribution/Results: Our method achieves a 3× FPS improvement over the original SAM2 while substantially reducing GPU memory usage. It is the first to enable stable, real-time video segmentation in resource-constrained surgical environments. The approach establishes a new paradigm for efficient and robust surgical video segmentation, directly supporting clinical decision-support systems.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Data transparency and provenance

Abstract

Surgical video segmentation is a critical task in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, the Segment Anything Model 2 (SAM2) framework has shown superior advancements in image and video segmentation. However, SAM2 struggles with efficiency due to the high computational demands of processing high-resolution images and complex and long-range temporal dynamics in surgical videos. To address these challenges, we introduce Surgical SAM 2 (SurgSAM-2), an advanced model to utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate real-time surgical video segmentation. The EFP mechanism dynamically manages the memory bank by selectively retaining only the most informative frames, reducing memory usage and computational cost while maintaining high segmentation accuracy. Our extensive experiments demonstrate that SurgSAM-2 significantly improves both efficiency and segmentation accuracy compared to the vanilla SAM2. Remarkably, SurgSAM-2 achieves a 3$ imes$ FPS compared with SAM2, while also delivering state-of-the-art performance after fine-tuning with lower-resolution data. These advancements establish SurgSAM-2 as a leading model for surgical video analysis, making real-time surgical video segmentation in resource-constrained environments a feasible reality.

Problem

• Enhance real-time surgical video segmentation efficiency
• Reduce computational cost in high-resolution video processing
• Improve segmentation accuracy with dynamic frame pruning

Innovation

• Efficient Frame Pruning reduces computational cost
• Dynamic memory bank management enhances efficiency
• Real-time segmentation with high accuracy achieved