VideoSEMA: a scalable and efficient Mamba-like attention for video understanding

2026-07-16

AI Summary

This work addresses the high computational complexity and poor scalability of existing models in high-resolution, long-sequence video understanding by proposing VideoSEMA. The method employs a decoupled spatiotemporal attention mechanism: spatially, it introduces a scalable and efficient Mamba-like attention (SEMA) that combines local windowed and global average attention; temporally, it utilizes softmax-based attention with support for sparse or dilated expansion strategies. Theoretically, under certain rank conditions, this decoupled design is equivalent to full spatiotemporal attention, balancing representational capacity and efficiency. Experiments demonstrate that VideoSEMA outperforms Vision Transformer and Mamba baselines of comparable scale on Kinetics-400 and Something-Something v2, and exhibits significantly milder accuracy degradation when image resolution scales from 224² to 1024², highlighting its superior scalability.

Abstract

We present for video understanding (classification) a split space-time attention model, VideoSEMA, consisting of a scalable and efficient Mamba-like attention (SEMA) block in space and a softmax temporal attention in time. In each frame, SEMA attention applies a local window attention in parallel with a global averaging in a Mamba macro-architecture, which is called Mamba-like. Under certain rank conditions, we prove that the computationally cheaper split space-time attention is equivalent to full space-time attention. On benchmark K400 data sets, VideoSEMA out-performs heavier vision transformer and Mamba models. On benchmark SSv2 data, VideoSEMA leads in top-1 accuracy among models of similar parameter sizes. As image resolution scales up from standard $224^2$ to $1024^2$ on K400 and without fine-tuning, VideoSEMA degrades much more gracefully than VideoMamba in accuracy. It is promising to extend VideoSEMA to longer videos with a dilated/sparse temporal attention.

Problem

• video understanding
• attention mechanism
• computational efficiency
• scalability
• video classification

Innovation

• Mamba-like attention
• split space-time attention
• scalable video modeling
• efficient video understanding
• local-global attention