Unlocking Positive Transfer in Incrementally Learning Surgical Instruments: A Self-reflection Hierarchical Prompt Framework

2026-04-03

AI Summary

This work addresses the lack of bidirectional knowledge transfer in incremental surgical instrument segmentation, which often leads to catastrophic forgetting and suboptimal learning efficiency. The authors propose a self-reflective hierarchical prompting framework that dynamically injects instrument-aware prompts into a frozen pre-trained model. Forward knowledge transfer—where prior knowledge aids learning of new classes—is achieved through a hierarchical prompt parsing tree, while backward transfer—enhancing representations of old classes with newly acquired knowledge—is realized via a self-reflective optimization mechanism based on a directed weighted graph propagation strategy. This approach is the first to systematically enable bidirectional knowledge transfer in incremental learning, offering architectural generality (compatible with both CNNs and Transformers) and strong scalability. It outperforms existing methods by over 5% and 11% on two public surgical video datasets, significantly improving segmentation performance for both novel and previously seen instruments.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Segmentation
• Machine Learning: Neuro-Symbolic Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

To continuously enhance model adaptability in surgical video scene parsing, recent studies incrementally update it to progressively learn to segment an increasing number of surgical instruments over time. However, prior works constantly overlooked the potential of positive forward knowledge transfer, i.e., how past knowledge could help learn new classes, and positive backward knowledge transfer, i.e., how learning new classes could help refine past knowledge. In this paper, we propose a self-reflection hierarchical prompt framework that unlocks the power of positive forward and backward knowledge transfer in class incremental segmentation, aiming to proficiently learn new instruments, improve existing skills of regular instruments, and avoid catastrophic forgetting of old instruments. Our framework is built on a frozen, pre-trained model that adaptively appends instrument-aware prompts for new classes throughout training episodes. To enable positive forward knowledge transfer, we organize instrument prompts into a hierarchical prompt parsing tree with the instrument-shared prompt partition as the root node, n-part-shared prompt partitions as intermediate nodes and instrument-distinct prompt partitions as leaf nodes, to expose the reusable historical knowledge for new classes to simplify their learning. Conversely, to encourage positive backward knowledge transfer, we conduct self-reflection refining on existing knowledge by directed-weighted graph propagation, examining the knowledge associations recorded in the tree to improve its representativeness without causing catastrophic forgetting. Our framework is applicable to both CNN-based models and advanced transformer-based foundation models, yielding more than 5% and 11% improvements over the competing methods on two public benchmarks respectively.

Problem

• positive transfer
• incremental learning
• surgical instrument segmentation
• catastrophic forgetting
• knowledge transfer

Innovation

• class-incremental segmentation
• hierarchical prompt
• positive knowledge transfer
• self-reflection
• catastrophic forgetting