UniSTFormer: Unified Spatio-Temporal Lightweight Transformer for Efficient Skeleton-Based Action Recognition

2025-08-12

AI Summary

Existing skeleton-based action recognition methods rely on stacked complex modules, resulting in excessive parameters, high computational cost, and poor scalability. To address this, we propose a lightweight unified spatiotemporal Transformer framework that integrates spatial and temporal modeling into a single spatiotemporal attention module, eliminating redundant architectural components. We further introduce a simplified multi-scale pooling fusion mechanism to jointly capture local joint-level details and global motion patterns. The entire architecture employs only one feature pathway, substantially reducing model complexity. Evaluated on standard benchmarks including NTU RGB+D, our method reduces parameter count by over 58% and FLOPs by over 60% compared to state-of-the-art approaches, while maintaining competitive accuracy. This achieves an exceptional trade-off between recognition performance and computational efficiency.

Technology Category

• Computer Vision: Motion & Tracking
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Skeleton-based action recognition (SAR) has achieved impressive progress with transformer architectures. However, existing methods often rely on complex module compositions and heavy designs, leading to increased parameter counts, high computational costs, and limited scalability. In this paper, we propose a unified spatio-temporal lightweight transformer framework that integrates spatial and temporal modeling within a single attention module, eliminating the need for separate temporal modeling blocks. This approach reduces redundant computations while preserving temporal awareness within the spatial modeling process. Furthermore, we introduce a simplified multi-scale pooling fusion module that combines local and global pooling pathways to enhance the model's ability to capture fine-grained local movements and overarching global motion patterns. Extensive experiments on benchmark datasets demonstrate that our lightweight model achieves a superior balance between accuracy and efficiency, reducing parameter complexity by over 58% and lowering computational cost by over 60% compared to state-of-the-art transformer-based baselines, while maintaining competitive recognition performance.

Problem

• Reducing parameter and computational costs in skeleton action recognition
• Integrating spatial and temporal modeling in single module
• Enhancing local and global motion pattern capture

Innovation

• Unified spatio-temporal lightweight transformer framework
• Single attention module for spatial and temporal modeling
• Simplified multi-scale pooling fusion module