SurgGoal: Rethinking Surgical Planning Evaluation via Goal-Satisfiability

2026-01-15

AI Summary

Current evaluation methods struggle to reliably assess the safety and correctness of vision-language models in surgical planning, particularly due to deficiencies in long-horizon reasoning and integration of procedural knowledge. This work proposes an expert rule–driven “phase-goal satisfiability” criterion as a high-precision meta-evaluation standard and constructs a multicenter surgical planning benchmark to systematically evaluate video large language models under diverse constraints. The study demonstrates that injecting structured knowledge significantly enhances model performance, whereas purely semantic guidance proves unreliable. It further reveals pervasive issues of perceptual errors and insufficient reasoning constraints in existing models. Critically, the work uncovers systematic biases in conventional sequence similarity metrics when distinguishing valid from invalid surgical plans, thereby establishing a new evaluation paradigm for safety-critical applications.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Surgical planning integrates visual perception, long-horizon reasoning, and procedural knowledge, yet it remains unclear whether current evaluation protocols reliably assess vision-language models (VLMs) in safety-critical settings. Motivated by a goal-oriented view of surgical planning, we define planning correctness via phase-goal satisfiability, where plan validity is determined by expert-defined surgical rules. Based on this definition, we introduce a multicentric meta-evaluation benchmark with valid procedural variations and invalid plans containing order and content errors. Using this benchmark, we show that sequence similarity metrics systematically misjudge planning quality, penalizing valid plans while failing to identify invalid ones. We therefore adopt a rule-based goal-satisfiability metric as a high-precision meta-evaluation reference to assess Video-LLMs under progressively constrained settings, revealing failures due to perception errors and under-constrained reasoning. Structural knowledge consistently improves performance, whereas semantic guidance alone is unreliable and benefits larger models only when combined with structural constraints.

Problem

• surgical planning
• evaluation protocol
• vision-language models
• goal satisfiability
• sequence similarity

Innovation

• goal-satisfiability
• surgical planning evaluation
• rule-based metric
• Video-LLMs
• structural knowledge