FRAME: Pre-Training Video Feature Representations via Anticipation and Memory

2025-06-05

AI Summary

Existing video encoders face bottlenecks in dense prediction tasks (e.g., object tracking, semantic segmentation): image encoders (e.g., DINO, CLIP) lack temporal modeling capability, while video models (e.g., VideoMAE) yield coarse spatial representations unsuitable for pixel-level alignment. To address this, we propose FRAME—a self-supervised frame encoder tailored for dense video understanding. FRAME takes historical and current RGB frames as input and jointly predicts DINO patch features for the current and future frames. It employs CLIP-based semantic space alignment and a lightweight temporal memory mechanism to produce token-level representations that are spatiotemporally consistent, spatially precise, and language-aligned. FRAME outperforms DINO, CLIP, and VideoMAE across seven datasets and six dense prediction tasks. Moreover, it generalizes effectively to cross-modal tasks such as video classification, achieving both compactness and strong transferability.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multimodal Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Dense video prediction tasks, such as object tracking and semantic segmentation, require video encoders that generate temporally consistent, spatially dense features for every frame. However, existing approaches fall short: image encoders like DINO or CLIP lack temporal awareness, while video models such as VideoMAE underperform compared to image encoders on dense prediction tasks. We address this gap with FRAME, a self-supervised video frame encoder tailored for dense video understanding. FRAME learns to predict current and future DINO patch features from past and present RGB frames, leading to spatially precise and temporally coherent representations. To our knowledge, FRAME is the first video encoder to leverage image-based models for dense prediction while outperforming them on tasks requiring fine-grained visual correspondence. As an auxiliary capability, FRAME aligns its class token with CLIP's semantic space, supporting language-driven tasks such as video classification. We evaluate FRAME across six dense prediction tasks on seven datasets, where it consistently outperforms image encoders and existing self-supervised video models. Despite its versatility, FRAME maintains a compact architecture suitable for a range of downstream applications.

Problem

• Develops a video encoder for dense prediction tasks
• Improves temporal consistency in video feature representation
• Outperforms image encoders in fine-grained visual correspondence

Innovation

• Predicts current and future DINO patch features
• Leverages image-based models for dense prediction
• Aligns class token with CLIP's semantic space