Training-Free Zero-Shot Temporal Action Detection with Vision-Language Models

2025-01-23

AI Summary

Existing zero-shot action recognition methods suffer from sensitivity to environmental variations and high computational overhead, limiting their practicality. This paper introduces the first training-free zero-shot temporal action detection framework, which directly leverages off-the-shelf vision-language models (ViLMs) to localize and classify unseen actions in untrimmed videos—eliminating domain shift and training costs entirely. Key contributions include: (1) a training-free paradigm; (2) a logarithmic decay-weighted intra- and inter-class contrastive scoring mechanism (LogOIC); (3) frequency-driven actionness calibration; and (4) prototype-centered sampling (PCS), a test-time adaptive strategy. Our method achieves substantial improvements over unsupervised state-of-the-art on THUMOS14 and ActivityNet-1.3, with inference speed 13× faster. When augmented with test-time adaptation (TTA), its performance approaches that of fully supervised methods.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Existing zero-shot temporal action detection (ZSTAD) methods predominantly use fully supervised or unsupervised strategies to recognize unseen activities. However, these training-based methods are prone to domain shifts and require high computational costs, which hinder their practical applicability in real-world scenarios. In this paper, unlike previous works, we propose a training-Free Zero-shot temporal Action Detection (FreeZAD) method, leveraging existing vision-language (ViL) models to directly classify and localize unseen activities within untrimmed videos without any additional fine-tuning or adaptation. We mitigate the need for explicit temporal modeling and reliance on pseudo-label quality by designing the LOGarithmic decay weighted Outer-Inner-Contrastive Score (LogOIC) and frequency-based Actionness Calibration. Furthermore, we introduce a test-time adaptation (TTA) strategy using Prototype-Centric Sampling (PCS) to expand FreeZAD, enabling ViL models to adapt more effectively for ZSTAD. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our training-free method outperforms state-of-the-art unsupervised methods while requiring only 1/13 of the runtime. When equipped with TTA, the enhanced method further narrows the gap with fully supervised methods.

Problem

• Zero-shot Action Recognition
• Environmental Variability
• Computational Cost

Innovation

• Zero-shot Action Detection
• Visual Language Models
• Prototype Center Sampling