Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition

2026-05-22

AI Summary

Existing open-vocabulary action recognition methods often lose fine-grained spatiotemporal details during global feature aggregation, hindering the modeling of nuanced action semantics. To address this, this work proposes SimVA, a framework that constructs a dense 4D spatiotemporal similarity volume to enable fine-grained alignment between local video patches and textual class descriptions. SimVA effectively transfers vision-language knowledge from CLIP by integrating spatial context modeling, motion-aware modulation, and Mamba-based temporal aggregation. Additionally, it introduces a class sampling strategy that facilitates scalable vocabulary expansion. The proposed method achieves state-of-the-art performance across zero-shot, few-shot, and base-to-novel action recognition benchmarks.

Abstract

Recent Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment, a process that obscures local patch information and fine-grained spatio-temporal cues. We propose Similarity Volume Aggregation (SimVA), a framework that constructs a dense 4D spatio-temporal similarity volume from patch-level visual-text similarities. SimVA constructs a spatio-temporal similarity volume over local video tokens and action classes, and employs class sampling to ensure similarity aggregation scalable to large vocabularies. The similarity volume is refined by spatial aggregation, which contextualizes local similarity patterns to improve intra-frame consistency. Motion-aware modulation further injects inter-frame variation cues, highlighting dynamically changing regions. Mamba-based temporal aggregation then models the evolution of class-conditioned similarity patterns across frames. By maintaining dense visual-text correspondence, SimVA effectively transfers CLIP to video action recognition, achieving competitive performance across zero-shot, few-shot, and base-to-novel benchmarks.

Problem

• Open-Vocabulary Action Recognition
• Spatio-Temporal Similarity
• Visual-Text Alignment
• Local Patch Information
• Fine-Grained Cues

Innovation

• Similarity Volume Aggregation
• Spatio-Temporal Modeling
• Open-Vocabulary Action Recognition
• Mamba-based Temporal Aggregation
• Visual-Text Alignment