How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment

2025-11-03

AI Summary

This work exposes critical causal reasoning deficiencies in current video generation models when applied to high-stakes surgical scenarios. To address the absence of domain-specific evaluation standards for surgical video generation, we introduce SurgVeo—the first expert-annotated benchmark—and propose the Surgical Plausibility Pyramid (SPP), a four-tiered evaluation framework assessing zero-shot generated videos across appearance, procedural execution, environmental feedback, and surgical intent. Leveraging Veo-3, we conduct experiments on laparoscopic and neurosurgical datasets, with multi-level assessments performed by four board-certified surgeons. Results demonstrate that while outputs achieve visual realism, models consistently fail at causal aspects: instrument manipulation logic, tissue interaction dynamics, and high-level surgical strategy intent—quantifying for the first time a “plausibility gap” in surgical video generation. This work establishes a new evaluation paradigm for medical video synthesis and highlights fundamental challenges in developing clinically reliable surgical world models.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Foundation models in video generation are demonstrating remarkable capabilities as potential world models for simulating the physical world. However, their application in high-stakes domains like surgery, which demand deep, specialized causal knowledge rather than general physical rules, remains a critical unexplored gap. To systematically address this challenge, we present SurgVeo, the first expert-curated benchmark for video generation model evaluation in surgery, and the Surgical Plausibility Pyramid (SPP), a novel, four-tiered framework tailored to assess model outputs from basic appearance to complex surgical strategy. On the basis of the SurgVeo benchmark, we task the advanced Veo-3 model with a zero-shot prediction task on surgical clips from laparoscopic and neurosurgical procedures. A panel of four board-certified surgeons evaluates the generated videos according to the SPP. Our results reveal a distinct "plausibility gap": while Veo-3 achieves exceptional Visual Perceptual Plausibility, it fails critically at higher levels of the SPP, including Instrument Operation Plausibility, Environment Feedback Plausibility, and Surgical Intent Plausibility. This work provides the first quantitative evidence of the chasm between visually convincing mimicry and causal understanding in surgical AI. Our findings from SurgVeo and the SPP establish a crucial foundation and roadmap for developing future models capable of navigating the complexities of specialized, real-world healthcare domains.

Problem

• Evaluating video generation models for surgical simulation accuracy
• Assessing AI's ability to understand specialized surgical causal knowledge
• Measuring the gap between visual realism and surgical strategy plausibility

Innovation

• Created SurgVeo benchmark for surgical video evaluation
• Introduced Surgical Plausibility Pyramid assessment framework
• Tested Veo-3 model on zero-shot surgical prediction