Large-scale Self-supervised Video Foundation Model for Intelligent Surgery

2025-06-03

AI Summary

Current AI-based surgical video understanding methods predominantly rely on self-supervised spatial representation learning but lack explicit temporal modeling, hindering comprehensive capture of dynamic surgical context. To address this, we propose SurgVISTA—the first large-scale self-supervised pretraining framework tailored for surgical videos—featuring (1) video-level joint spatiotemporal representation learning and (2) a surgical-expert-guided image-level knowledge distillation mechanism. SurgVISTA is pretrained via reconstruction on 3,650 surgical videos (3.55M frames) and evaluated on a novel multi-task benchmark comprising 13 datasets spanning six surgical procedures and four task categories. Extensive experiments demonstrate that SurgVISTA consistently outperforms existing models—both in natural-domain and surgical-domain settings—achieving significant gains in clinically relevant surgical video understanding tasks.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Computer-Assisted Intervention (CAI) has the potential to revolutionize modern surgery, with surgical scene understanding serving as a critical component in supporting decision-making, improving procedural efficacy, and ensuring intraoperative safety. While existing AI-driven approaches alleviate annotation burdens via self-supervised spatial representation learning, their lack of explicit temporal modeling during pre-training fundamentally restricts the capture of dynamic surgical contexts, resulting in incomplete spatiotemporal understanding. In this work, we introduce the first video-level surgical pre-training framework that enables joint spatiotemporal representation learning from large-scale surgical video data. To achieve this, we constructed a large-scale surgical video dataset comprising 3,650 videos and approximately 3.55 million frames, spanning more than 20 surgical procedures and over 10 anatomical structures. Building upon this dataset, we propose SurgVISTA (Surgical Video-level Spatial-Temporal Architecture), a reconstruction-based pre-training method that captures intricate spatial structures and temporal dynamics through joint spatiotemporal modeling. Additionally, SurgVISTA incorporates image-level knowledge distillation guided by a surgery-specific expert to enhance the learning of fine-grained anatomical and semantic features. To validate its effectiveness, we established a comprehensive benchmark comprising 13 video-level datasets spanning six surgical procedures across four tasks. Extensive experiments demonstrate that SurgVISTA consistently outperforms both natural- and surgical-domain pre-trained models, demonstrating strong potential to advance intelligent surgical systems in clinically meaningful scenarios.

Problem

• Lack of temporal modeling in surgical AI limits dynamic context capture
• Need for joint spatiotemporal learning in surgical video pre-training
• Incomplete understanding of surgical scenes affects decision-making and safety

Innovation

• Joint spatiotemporal representation learning from surgical videos
• Reconstruction-based pre-training for spatial-temporal modeling
• Surgery-specific knowledge distillation for fine-grained features