PLOT-TAL - Prompt Learning with Optimal Transport for Few-Shot Temporal Action Localization

2024-03-27

arXiv.org

AI Summary

Existing few-shot temporal action localization (TAL) methods rely on single-prompt tuning, rendering action representations non-discriminative and temporal boundaries inaccurate due to sparse annotations. To address this, we propose a multi-prompt learning framework—the first to incorporate optimal transport (Wasserstein distance) into few-shot TAL. Our approach models intrinsic action characteristics via a learnable, diverse prompt set, enabling dynamic prompt–feature matching and distributionally robust alignment between prompts and action features. The framework integrates multi-prompt generation, Transformer-based temporal modeling, and a meta-training paradigm. Evaluated on THUMOS-14 and EpicKitchens-100 under the 5-shot setting, our method achieves a 9.2% mAP improvement over state-of-the-art methods. It significantly enhances localization accuracy, generalization across viewpoints, backgrounds, and objects, and cross-domain transferability.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Temporal Planning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This paper introduces a novel approach to temporal action localization (TAL) in few-shot learning. Our work addresses the inherent limitations of conventional single-prompt learning methods that often lead to overfitting due to the inability to generalize across varying contexts in real-world videos. Recognizing the diversity of camera views, backgrounds, and objects in videos, we propose a multi-prompt learning framework enhanced with optimal transport. This design allows the model to learn a set of diverse prompts for each action, capturing general characteristics more effectively and distributing the representation to mitigate the risk of overfitting. Furthermore, by employing optimal transport theory, we efficiently align these prompts with action features, optimizing for a comprehensive representation that adapts to the multifaceted nature of video data. Our experiments demonstrate significant improvements in action localization accuracy and robustness in few-shot settings on the standard challenging datasets of THUMOS-14 and EpicKitchens100, highlighting the efficacy of our multi-prompt optimal transport approach in overcoming the challenges of conventional few-shot TAL methods.

Problem

• Addresses imprecise temporal boundaries in few-shot action localization
• Overcomes non-discriminative representations from sparse data
• Enhances generalization through multi-prompt ensemble specialization

Innovation

• Multi-prompt ensembles for action representation
• Optimal Transport for temporal feature alignment
• Learning distributed compositional sub-event representations