Surg-3M: A Dataset and Foundation Model for Perception in Surgical Settings

2025-03-25

AI Summary

Small-scale conventional surgical vision datasets hinder progress in surgical scene understanding and autonomous robotic surgery. To address this, we introduce Surg-3M—the first large-scale open-source surgical vision dataset—comprising 4K videos and over 3 million annotated images. Leveraging Surg-3M, we propose SurgFM, a self-supervised foundational model tailored for surgical vision. SurgFM innovatively integrates a ConvNeXt backbone with the DINO framework and incorporates an enhanced knowledge distillation mechanism to improve cross-task generalization. Evaluated on three core surgical tasks—surgical phase recognition, surgical action recognition, and surgical instrument detection—SurgFM consistently surpasses state-of-the-art methods: it achieves absolute improvements of 3.9–8.9 percentage points in Jaccard index and 3.1/4.6 percentage points in mAP, respectively. Remarkably, SurgFM attains optimal performance using only 50% of the training data.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Advancements in computer-assisted surgical procedures heavily rely on accurate visual data interpretation from camera systems used during surgeries. Traditional open-access datasets focusing on surgical procedures are often limited by their small size, typically consisting of fewer than 100 videos with less than 100K images. To address these constraints, a new dataset called Surg-3M has been compiled using a novel aggregation pipeline that collects high-resolution videos from online sources. Featuring an extensive collection of over 4K surgical videos and more than 3 million high-quality images from multiple procedure types, Surg-3M offers a comprehensive resource surpassing existing alternatives in size and scope, including two novel tasks. To demonstrate the effectiveness of this dataset, we present SurgFM, a self-supervised foundation model pretrained on Surg-3M that achieves impressive results in downstream tasks such as surgical phase recognition, action recognition, and tool presence detection. Combining key components from ConvNeXt, DINO, and an innovative augmented distillation method, SurgFM exhibits exceptional performance compared to specialist architectures across various benchmarks. Our experimental results show that SurgFM outperforms state-of-the-art models in multiple downstream tasks, including significant gains in surgical phase recognition (+8.9pp, +4.7pp, and +3.9pp of Jaccard in AutoLaparo, M2CAI16, and Cholec80), action recognition (+3.1pp of mAP in CholecT50) and tool presence detection (+4.6pp of mAP in Cholec80). Moreover, even when using only half of the data, SurgFM outperforms state-of-the-art models in AutoLaparo and achieves state-of-the-art performance in Cholec80. Both Surg-3M and SurgFM have significant potential to accelerate progress towards developing autonomous robotic surgery systems.

Problem

• Limited size of traditional surgical datasets hinders computer-assisted surgery.
• Lack of comprehensive datasets for multiple surgical procedure types.
• Need for improved foundation models for surgical perception tasks.

Innovation

• Large-scale dataset Surg-3M with 4K videos
• Self-supervised foundation model SurgFM
• Combines ConvNeXt, DINO, and augmented distillation