🤖 AI Summary
To address the high computational cost of temporal modeling and insufficient exploitation of global temporal context in video representation learning, this paper proposes the Supervised Contrastive Frame Aggregation (SCFA) framework. SCFA spatially arranges multiple frames into a single pseudo-image, enabling efficient reuse of lightweight image backbones (e.g., ResNet-50) and avoiding the prohibitive computational overhead of video Transformers. A supervised contrastive loss—guided by class labels—is introduced, and natural positive sample pairs are constructed via multi-temporal sampling within the same video to explicitly encode global temporal structure. The framework supports end-to-end video classification and captioning. On Penn Action and HMDB51, SCFA achieves 76% and 48% top-1 accuracy, respectively—substantially outperforming ViViT (43%/37%)—while significantly improving both training and inference efficiency.
📝 Abstract
We propose a supervised contrastive learning framework for video representation learning that leverages temporally global context. We introduce a video to image aggregation strategy that spatially arranges multiple frames from each video into a single input image. This design enables the use of pre trained convolutional neural network backbones such as ResNet50 and avoids the computational overhead of complex video transformer models. We then design a contrastive learning objective that directly compares pairwise projections generated by the model. Positive pairs are defined as projections from videos sharing the same label while all other projections are treated as negatives. Multiple natural views of the same video are created using different temporal frame samplings from the same underlying video. Rather than relying on data augmentation these frame level variations produce diverse positive samples with global context and reduce overfitting. Experiments on the Penn Action and HMDB51 datasets demonstrate that the proposed method outperforms existing approaches in classification accuracy while requiring fewer computational resources. The proposed Supervised Contrastive Frame Aggregation method learns effective video representations in both supervised and self supervised settings and supports video based tasks such as classification and captioning. The method achieves seventy six percent classification accuracy on Penn Action compared to forty three percent achieved by ViVIT and forty eight percent accuracy on HMDB51 compared to thirty seven percent achieved by ViVIT.