Efficient Frame Extraction: A Novel Approach Through Frame Similarity and Surgical Tool Tracking for Video Segmentation

2025-01-19

AI Summary

To address the inefficiency and automation challenges in AI-based analysis of lengthy, high-volume surgical videos, this paper proposes Kinematic-Adaptive Keyframe Recognition (KAFR). KAFR innovatively integrates instrument kinematic similarity modeling with an adaptive frame sampling paradigm, overcoming limitations of conventional uniform sampling or unsupervised compression to precisely identify semantically relevant frames. Technically, it unifies YOLOv8-based instrument detection, motion position/velocity difference modeling, and the X3D spatiotemporal convolutional network for end-to-end segmentation and classification. Evaluated on two multi-year, multi-institutional clinical datasets—GJ (2017–2021) and PJ (2011–2022)—KAFR achieves a 10× frame compression ratio while improving segmentation mIoU by 4.32 percentage points (from 0.749 to 0.7814), significantly enhancing both computational efficiency and analytical accuracy.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

The interest in leveraging Artificial Intelligence (AI) for surgical procedures to automate analysis has witnessed a significant surge in recent years. One of the primary tools for recording surgical procedures and conducting subsequent analyses, such as performance assessment, is through videos. However, these operative videos tend to be notably lengthy compared to other fields, spanning from thirty minutes to several hours, which poses a challenge for AI models to effectively learn from them. Despite this challenge, the foreseeable increase in the volume of such videos in the near future necessitates the development and implementation of innovative techniques to tackle this issue effectively. In this article, we propose a novel technique called Kinematics Adaptive Frame Recognition (KAFR) that can efficiently eliminate redundant frames to reduce dataset size and computation time while retaining useful frames to improve accuracy. Specifically, we compute the similarity between consecutive frames by tracking the movement of surgical tools. Our approach follows these steps: i) Tracking phase: a YOLOv8 model is utilized to detect tools presented in the scene, ii) Similarity phase: Similarities between consecutive frames are computed by estimating variation in the spatial positions and velocities of the tools, iii) Classification phase: A X3D CNN is trained to classify segmentation. We evaluate the effectiveness of our approach by analyzing datasets obtained through retrospective reviews of cases at two referral centers. The Gastrojejunostomy (GJ) dataset covers procedures performed between 2017 to 2021, while the Pancreaticojejunostomy (PJ) dataset spans from 2011 to 2022 at the same centers. By adaptively selecting relevant frames, we achieve a tenfold reduction in the number of frames while improving accuracy by 4.32% (from 0.749 to 0.7814).

Problem

• Surgical Video Analysis
• Data Reduction
• AI Efficiency

Innovation

• Kinematic Adaptive Frame Recognition
• Surgical Tool Dynamics
• AI Efficiency Enhancement