🤖 AI Summary
This work addresses the limitations of existing action detection methods, which are often constrained by single-view training data and struggle to model fine-grained temporal dependencies. To overcome these challenges, the authors propose a two-stage action detection framework. In the first stage, virtual viewpoint augmentation is employed during training to extract viewpoint-invariant motion features. The second stage introduces a multi-scale temporal encoder based on a selective state space model, effectively integrating information across multiple viewpoints and temporal scales. This approach represents the first integration of virtual viewpoint augmentation with selective state space modeling for sequence representation. It achieves consistent and significant improvements over state-of-the-art methods across all splits of the PKU-MMD and BABEL benchmarks, while simultaneously enhancing viewpoint robustness and global temporal consistency.
📝 Abstract
Viewpoint change invariance and action temporal consistency are critical aspects for the effective deployment of human action detection of untrimmed videos. Existing appearance-based video detection methods often struggle with limited viewpoint diversity during training, while motion-based detection approaches frequently fail to model fine-grained temporal relationships across consecutive motion windows. This paper introduces a novel two-stage action detection approach designed to improve both view-invariance and global temporal coherence properties. In the first stage, we extract motion features from augmented virtual viewpoints, solely used at training. Then, the second stage introduces a new view-invariant, multi-scale temporal encoder based on selective state-space sequence modelling to aggregate information across viewpoints and time scales. Experiments on PKU-MMD and BABEL benchmarks demonstrate that this approach significantly outperforms state-of-the-art methods in all considered splits. Code and trained models are available at: https://icb-vision-ai.github.io/HydraView-TAD