Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detection

2026-03-25

AI Summary

This work addresses the challenge in open-vocabulary temporal action detection (OV-TAD) where existing methods rely on global alignment between label-level semantics and visual features, limiting their ability to transfer temporally consistent knowledge from seen to unseen action categories. To overcome this, the authors propose a phase decomposition and alignment framework that leverages chain-of-thought prompting (CoT-Prompting) from large language models to automatically decompose action labels into stage-level semantic descriptions. This decomposition enables fine-grained visual-textual alignment through two key components: text-guided foreground filtering (TIF) and adaptive phase alignment (APA). The resulting model significantly enhances generalization to unseen actions and achieves state-of-the-art performance on two OV-TAD benchmarks, demonstrating its effectiveness and superiority.

Technology Category

• Computer Vision: Language and Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-Vocabulary Temporal Action Detection (OV-TAD) aims to classify and localize action segments in untrimmed videos for unseen categories. Previous methods rely solely on global alignment between label-level semantics and visual features, which is insufficient to transfer temporal consistent visual knowledge from seen to unseen classes. To address this, we propose a Phase-wise Decomposition and Alignment (PDA) framework, which enables fine-grained action pattern learning for effective prior knowledge transfer. Specifically, we first introduce the CoT-Prompting Semantic Decomposition (CSD) module, which leverages the chain-of-thought (CoT) reasoning ability of large language models to automatically decompose action labels into coherent phase-level descriptions, emulating human cognitive processes. Then, Text-infused Foreground Filtering (TIF) module is introduced to adaptively filter action-relevant segments for each phase leveraging phase-wise semantic cues, producing semantically aligned visual representations. Furthermore, we propose the Adaptive Phase-wise Alignment (APA) module to perform phase-level visual-textual matching, and adaptively aggregates alignment results across phases for final prediction. This adaptive phase-wise alignment facilitates the capture of transferable action patterns and significantly enhances generalization to unseen actions. Extensive experiments on two OV-TAD benchmarks demonstrated the superiority of the proposed method.

Problem

• Open-Vocabulary Temporal Action Detection
• knowledge transfer
• temporal action localization
• unseen action categories
• visual-textual alignment

Innovation

• Chain-of-Thought Prompting
• Phase-wise Alignment
• Open-Vocabulary Temporal Action Detection
• Semantic Decomposition
• Visual-Textual Matching