SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery

2026-06-24

AI Summary

This work addresses the limited scale and narrow scope of existing surgical video–language datasets, which predominantly exclude open procedures and thereby hinder the generalization and reasoning capabilities of AI models. To overcome this, the authors introduce SurgAtlas, the largest multimodal surgical dataset to date, encompassing 18 specialties and over 5,000 distinct procedures, with 15,291 videos totaling 2,391 hours—marking the first large-scale integration of both open and minimally invasive surgeries. The study proposes a hierarchical annotation framework and an expert-validated visual question answering benchmark, leveraging an LLM-based automated multilevel annotation pipeline with grounding verification. Through a two-stage fine-tuning strategy on Qwen3-VL-8B, the model achieves state-of-the-art performance in surgical phase recognition, triplet detection, and reasoning-based question answering, offering a high-quality pretraining resource for foundational multimodal surgical models.

Abstract

We introduce SurgAtlas, the largest surgical video-language dataset to date, comprising 15,291 videos (2,391 hours) spanning 18 surgical specialties and over 5,000 procedure types, sourced entirely from publicly available YouTube content. SurgAtlas is also the first surgical video-language dataset to include open surgery at scale, with 6,182 open procedure videos alongside over 9,000 minimally invasive recordings, and the first to establish standardized benchmarks for open-surgery video understanding. We additionally provide an expert-validated subset with verified visual question-answer pairs across diverse open and minimally invasive procedures, serving as a clinically grounded benchmark for surgical reasoning. Compared with existing surgical video-language datasets, SurgAtlas provides one of the most diverse annotation schemas, combining segment-level captions, step- and phase-level descriptions, video-level surgical descriptions, and reasoning-oriented question-answer pairs organized within a hierarchical taxonomy. These annotations are constructed through an automated multi-tier pipeline with LLM-based enrichment and a staged VQA generation framework with explicit groundedness verification. The scale and diversity of SurgAtlas enable training surgical foundation models with broad procedural coverage: we finetune Qwen3-VL-8B through a two-stage captioning-then-instruction pipeline and achieve competitive or state-of-the-art results on multiple established surgical benchmarks, including phase recognition, triplet detection, and reasoning question answering. More broadly, SurgAtlas provides a large native public video corpus that can support future large-scale pretraining of multimodal surgical AI systems and contribute to the development of next-generation foundation models for surgery.

Problem

• surgical video-language dataset
• open surgery
• minimally invasive surgery
• video understanding benchmark
• surgical reasoning

Innovation

• surgical video-language dataset
• open surgery benchmarking
• multimodal foundation model
• LLM-enhanced annotation
• grounded visual question answering