Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions

2026-06-25

AI Summary

This work addresses the challenge of mismatched spatiotemporal annotation granularities in surgical video multitask learning, where temporal tasks require dense frame-level supervision while spatial tasks are annotated only sparsely on keyframes, hindering effective shared representation learning. To overcome this, the authors propose FAROS, a novel framework that, for the first time in surgical settings, generates temporally consistent pseudo-labels robust to occlusions, smoke, and motion blur. FAROS leverages zero-shot segmentation and optical flow–guided mask propagation to interpolate dense pseudo-labels from sparse keyframe annotations, which are then integrated into a unified Transformer-based multitask model for joint optimization. Experiments demonstrate the robustness of the interpolation strategy on DAVIS 2017 and show significant improvements in multitask representation learning and holistic scene understanding on the GraSP, MISAW, and AutoLaparo datasets.

Abstract

Effective multi-task learning for surgical scene understanding is fundamentally hindered by annotation granularity mismatch; temporal workflow tasks such as phase recognition, step recognition and anticipation benefit from dense frame-level supervision, whereas pixel-level spatial tasks including instrument segmentation and action recognition are only sparsely annotated on selected keyframes due to prohibitive labeling costs. This supervision imbalance undermines shared representation learning and limits joint optimization across heterogeneous surgical tasks. To address this, we propose Flow-guided Annotation for Robust Operating Scenes (FAROS), a flow-guided label interpolation framework, that combines zero-shot segmentation-based mask propagation with optical flow estimation to overcome the limitations of appearance-based propagation under challenging surgical conditions such as occlusion, smoke, and motion blur, generating temporally consistent dense pseudo labels from sparse keyframe annotations. The densified instrument masks and action labels are integrated into a unified Transformer-based multi-task framework that jointly learns surgical phase recognition, step recognition, anticipation, instrument segmentation, and action recognition, enabling balanced optimization between dense temporal supervision and sparse spatial supervision. The label interpolation quality of FAROS is first validated on the DAVIS 2017 benchmark under a sparse ground-truth protocol, confirming robust propagation beyond the surgical domain. Extensive experiments on GraSP, MISAW, and AutoLaparo benchmarks further demonstrate that FAROS significantly improves cross-task representation learning and enhances holistic surgical scene understanding performance across spatio-temporal tasks.

Problem

• annotation granularity mismatch
• surgical multi-task learning
• temporal workflow tasks
• sparse spatial annotation
• supervision imbalance

Innovation

• label interpolation
• optical flow
• multi-task learning
• surgical scene understanding
• temporal consistency