A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Ranking

2025-11-21

AI Summary

Existing self-supervised video learning methods struggle to model the inherent temporal structure of procedural activities—such as surgery and cooking—and fail to distinguish forward sequences from time-reversed ones. To address this, we propose PL-Stitch, the first framework to incorporate the Plackett–Luce ranking model into self-supervised video learning, jointly capturing global procedural order and fine-grained spatiotemporal dependencies. PL-Stitch introduces a spatiotemporal jigsaw loss and a contrastive representation learning objective, enabling implicit learning of causal and phase-wise relationships among actions via frame-level sampling and ranking optimization. Evaluated on five surgical and cooking benchmarks, PL-Stitch achieves state-of-the-art performance: it improves k-NN accuracy for surgical phase recognition by 11.4 percentage points and linear-probe accuracy for cooking action segmentation by 5.7 percentage points—demonstrating significantly enhanced structured representation capability for procedural activities.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Machine Learning: Structured Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Personalized content ranking and presentation

Abstract

Procedural activities, ranging from routine cooking to complex surgical operations, are highly structured as a set of actions conducted in a specific temporal order. Despite their success on static images and short clips, current self-supervised learning methods often overlook the procedural nature that underpins such activities. We expose the lack of procedural awareness in current SSL methods with a motivating experiment: models pretrained on forward and time-reversed sequences produce highly similar features, confirming that their representations are blind to the underlying procedural order. To address this shortcoming, we propose PL-Stitch, a self-supervised framework that harnesses the inherent temporal order of video frames as a powerful supervisory signal. Our approach integrates two novel probabilistic objectives based on the Plackett-Luce (PL) model. The primary PL objective trains the model to sort sampled frames chronologically, compelling it to learn the global workflow progression. The secondary objective, a spatio-temporal jigsaw loss, complements the learning by capturing fine-grained, cross-frame object correlations. Our approach consistently achieves superior performance across five surgical and cooking benchmarks. Specifically, PL-Stitch yields significant gains in surgical phase recognition (e.g., +11.4 pp k-NN accuracy on Cholec80) and cooking action segmentation (e.g., +5.7 pp linear probing accuracy on Breakfast), demonstrating its effectiveness for procedural video representation learning.

Problem

• Current self-supervised methods ignore procedural temporal order in videos
• Models fail to distinguish forward and reversed video sequence order
• Proposed framework learns workflow progression through temporal sorting objectives

Innovation

• Self-supervised framework using temporal order
• Plackett-Luce model for chronological frame sorting
• Spatio-temporal jigsaw loss capturing object correlations