Momentum-Guided Semantic Forecasting (MoFore) for Self-Supervised Video Representation Learning

2026-06-08

AI Summary

This work addresses the challenge of learning video representations that are both temporally coherent and semantically meaningful in the absence of labeled data. To this end, the authors propose a momentum-guided semantic prediction framework that performs self-supervised learning by predicting future latent embeddings across randomly sampled temporal intervals, thereby eliminating reliance on pixel-level reconstruction or task-specific alignment. A contrastive regularization term is further introduced to enhance temporal consistency and prevent representation collapse. Experimental results demonstrate that the learned embedding space on UCF101 not only exhibits strong temporal stability but also explicitly captures action semantics and motion patterns, reflecting a superior capacity for structured representation learning.

Abstract

Self-supervised video representation learning has recently advanced through contrastive learning, masked reconstruction, and predictive representation learning. Reconstruction-based approaches such as MAE and VideoMAE learn representations by recovering masked visual content \cite{he2022mae,tong2022videomae}, while contrastive methods such as CLIP learn semantically meaningful embedding spaces through representation alignment \cite{radford2021clip}. In this work, we introduce a Momentum-Guided Semantic Forecasting framework (MoFore) for self-supervised video representation learning. Instead of optimizing for pixel-level reconstruction or task-specific semantic alignment, the proposed method learns temporally predictive video representations by forecasting future latent embeddings from temporally distant context clips. To improve robustness across temporal scales, we further introduce randomized temporal-gap forecasting during training. The framework combines predictive latent forecasting with contrastive regularization to encourage temporal consistency while preventing representation collapse. Experiments on the UCF101 dataset demonstrate that the proposed framework learns temporally consistent and semantically meaningful video representations without using action labels during training. Quantitative analysis shows strong temporal stability and emergent category-level structure in the learned embedding space, while qualitative retrieval experiments reveal motion-aware organization across related activities. Overall, the results suggest that long-range latent forecasting provides an effective and computationally efficient approach for self-supervised video representation learning without relying on reconstruction-based objectives.

Problem

• self-supervised learning
• video representation learning
• temporal consistency
• semantic forecasting
• latent forecasting

Innovation

• Momentum-Guided Semantic Forecasting
• self-supervised video representation learning
• temporal predictive modeling
• latent forecasting
• contrastive regularization