🤖 AI Summary
Real-time video analysis faces a fundamental trade-off between spatiotemporal modeling accuracy and inference efficiency, particularly under resource constraints. To address this, we propose a unified multi-task framework that jointly performs action recognition and object tracking. Our approach employs a hierarchical attention mechanism to adaptively focus on temporally salient spatial regions and introduces parallel sequence modeling to enhance computational efficiency. By integrating advanced spatiotemporal representation learning with lightweight co-optimization strategies, the method achieves state-of-the-art performance: +3.2% and +2.8% top-1 accuracy on UCF-101 and HMDB-51 for action recognition, +2.8% MOTA on MOT17 for multi-object tracking, and a 40% speedup in end-to-end inference latency—significantly outperforming existing real-time approaches.
📝 Abstract
Real-time video analysis remains a challenging problem in computer vision, requiring efficient processing of both spatial and temporal information while maintaining computational efficiency. Existing approaches often struggle to balance accuracy and speed, particularly in resource-constrained environments. In this work, we present a unified framework that leverages advanced spatial-temporal modeling techniques for simultaneous action recognition and object tracking. Our approach builds upon recent advances in parallel sequence modeling and introduces a novel hierarchical attention mechanism that adaptively focuses on relevant spatial regions across temporal sequences. We demonstrate that our method achieves state-of-the-art performance on standard benchmarks while maintaining real-time inference speeds. Extensive experiments on UCF-101, HMDB-51, and MOT17 datasets show improvements of 3.2% in action recognition accuracy and 2.8% in tracking precision compared to existing methods, with 40% faster inference time.