An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models

2026-04-01

AI Summary

Existing surgical vision-language datasets struggle to model complex spatiotemporal dynamics, limiting fine-grained understanding by current models. To address this, this work proposes SurgSTU-Pipeline, a deterministic spatiotemporal continuity filtering framework that automatically constructs high-quality, fine-grained spatiotemporal question-answer pairs from publicly available surgical videos—without requiring manual annotations or reliance on large language models. Using this pipeline, the authors curate the SurgSTU dataset, comprising 7,515 video clips and 150,000 question-answer pairs. Experiments demonstrate that fine-tuning and in-context learning with SurgSTU significantly enhance the performance of vision-language models on surgical video spatiotemporal reasoning tasks, achieving state-of-the-art results.

Technology Category

• Computer Vision: Language and Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Surgical video understanding is a crucial prerequisite for advancing Computer-Assisted Surgery. While vision-language models (VLMs) have recently been applied to the surgical domain, existing surgical vision-language datasets lack in capturing and evaluating complex, interleaved spatial-temporal dynamics. Creating large scale datasets that accurately represent fine-grained spatial-temporal relationships in surgical videos is challenging due to costly manual annotations or error-prone generation using large language models. To address this gap, we introduce the SurgSTU-Pipeline, a deterministic generation pipeline featuring temporal and spatial continuity filtering to reliably create surgical datasets for fine-grained spatial-temporal multimodal understanding. Applying this pipeline to publicly available surgical datasets, we create the SurgSTU dataset, comprising 7515 video clips densely extended with 150k fine-grained spatial-temporal question-answer samples. Our comprehensive evaluation shows that while state-of-the-art generalist VLMs struggle in zero-shot settings, their spatial-temporal capabilities can be improved through in-context learning. A fine-tuned VLM on the SurgSTU training dataset achieves highest performance among all spatial-temporal tasks, validating the dataset's efficacy to improve spatial-temporal understanding of VLMs in surgical videos. Code will be made publicly available.

Problem

• surgical video understanding
• vision-language models
• spatial-temporal dynamics
• fine-grained annotation
• multimodal dataset

Innovation

• SurgSTU-Pipeline
• spatial-temporal understanding
• surgical video dataset
• vision-language models
• deterministic generation