Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)

2025-11-09

AI Summary

Current robotic/Computer-Assisted Minimally Invasive Surgery (RAMIS) research lacks large-scale, high-quality datasets that simultaneously support multi-center evaluation, temporal modeling, and multi-task annotation; existing resources are typically confined to single-task, frame-wise, or single-center settings. Method: We introduce the first multi-center laparoscopic cholecystectomy video dataset comprising eight complete procedures, uniquely providing synchronized frame-level surgical phase labels, instrument keypoint pose estimates, and instance segmentation masks. All data were acquired under strict clinical protocols and meticulously annotated. Contribution/Results: The dataset explicitly enables temporal context modeling and end-to-end joint analysis across tasks. It is publicly released on Zenodo and serves as the official benchmark for the MICCAI 2024 EndoVis PhaKIR Challenge, thereby addressing critical gaps in task integration and generalizability for RAMIS research.

Technology Category

• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision

Application Category

• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Robotic- and computer-assisted minimally invasive surgery (RAMIS) is increasingly relying on computer vision methods for reliable instrument recognition and surgical workflow understanding. Developing such systems often requires large, well-annotated datasets, but existing resources often address isolated tasks, neglect temporal dependencies, or lack multi-center variability. We present the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) dataset, comprising eight complete laparoscopic cholecystectomy videos recorded at three medical centers. The dataset provides frame-level annotations for three interconnected tasks: surgical phase recognition (485,875 frames), instrument keypoint estimation (19,435 frames), and instrument instance segmentation (19,435 frames). PhaKIR is, to our knowledge, the first multi-institutional dataset to jointly provide phase labels, instrument pose information, and pixel-accurate instrument segmentations, while also enabling the exploitation of temporal context since full surgical procedure sequences are available. It served as the basis for the PhaKIR Challenge as part of the Endoscopic Vision (EndoVis) Challenge at MICCAI 2024 to benchmark methods in surgical scene understanding, thereby further validating the dataset's quality and relevance. The dataset is publicly available upon request via the Zenodo platform.

Problem

• Develops multi-task surgical vision system for laparoscopic procedures
• Addresses lack of integrated datasets with temporal dependencies
• Provides joint phase recognition and instrument pose estimation

Innovation

• Multi-center dataset for laparoscopic surgery analysis
• Joint annotations for phases, keypoints and instrument segmentation
• Full surgical sequences enabling temporal context exploitation