Skeleton-based Zero-Shot Spatio-Temporal Action Localization via Weakly-Supervised Pretraining

2026-08-26

AI Summary

本文提出了一种基于弱监督预训练的骨架-语言特征池切换方法，解决了零样本时空动作定位中的标注成本高问题。

Abstract

We propose a novel pretraining strategy for skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances while overcoming high annotation costs for training via new target actions and pretraining using large-scale action scenery datasets. Specifically, our approach, termed Skeleton-Language feature Pooling Switching, introduces a weakly-supervised vision-language pretraining mechanism. This mechanism transitions pooling kernels from pretraining, which aggregates skeleton features at the video level and aligns them with each video's known action text embeddings, to the inference phase that computes instance-level features without training via target actions. Furthermore, we propose Scene-Mixed Discriminative Contrastive Learning to distinguish actions at the instance level within the combined scene through the MIL framework. Our experiments on four public spatio-temporal action localization and classification datasets demonstrate that the proposed method effectively addresses annotation limitations.

Problem

• Skeleton-based
• Zero-Shot
• Spatio-Temporal Action Localization
• Annotation Costs
• Weakly-Supervised

Innovation

• Skeleton-Language feature Pooling Switching
• Weakly-Supervised Pretraining
• Scene-Mixed Discriminative Contrastive Learning