Surgical Visual Understanding (SurgVU) Dataset

2025-01-16

AI Summary

High-quality, multi-task annotated video datasets are scarce in robot-assisted surgery, hindering the development of robust surgical AI models. Method: This work introduces the first large-scale, open-source surgical video dataset featuring synchronized, fine-grained temporal annotations across three complementary semantic dimensions—surgical instruments, anatomical structures, and procedural phases—curated from real clinical robotic surgery recordings by expert annotators. A unified, high-fidelity hierarchical taxonomy is established to support diverse vision-language tasks. Contribution/Results: The dataset comprises thousands of complete surgical videos with comprehensive annotations, enabling benchmarking for computer vision, temporal modeling, and multimodal learning. It has been validated on foundational tasks such as surgical tool detection and has become a widely adopted standard benchmark in surgical AI, effectively addressing critical gaps in standardized, multi-task evaluation frameworks for surgical data science.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Owing to recent advances in machine learning and the ability to harvest large amounts of data during robotic-assisted surgeries, surgical data science is ripe for foundational work. We present a large dataset of surgical videos and their accompanying labels for this purpose. We describe how the data was collected and some of its unique attributes. Multiple example problems are outlined. Although the dataset was curated for a particular set of scientific challenges (in an accompanying paper), it is general enough to be used for a broad range machine learning questions. Our hope is that this dataset exposes the larger machine learning community to the challenging problems within surgical data science, and becomes a touchstone for future research. The videos are available at https://storage.googleapis.com/isi-surgvu/surgvu24_videos_only.zip, the labels at https://storage.googleapis.com/isi-surgvu/surgvu24_labels_updated_v2.zip, and a validation set for tool detection problem at https://storage.googleapis.com/isi-surgvu/cat1_test_set_public.zip.

Problem

• Robot-Assisted Surgery
• Machine Learning
• Surgical Data Science

Innovation

• SurgVU
• Surgical Data Science
• Machine Learning