CrossVideoMAE: Self-Supervised Image-Video Representation Learning with Masked Autoencoders

2025-02-08

AI Summary

Existing video masked autoencoders (MAEs) focus on generic spatiotemporal pattern modeling while neglecting fine-grained action semantics—such as interaction timing and object relations—and typically train video and image modalities in isolation, leading to coarse-grained semantic representations and decoupled spatiotemporal-semantic learning. To address this, we propose the first end-to-end self-supervised cross-modal MAE framework that jointly models video sequences and sampled frames. It performs contrastive masked reconstruction within a feature-invariant space, leveraging intra- and inter-modal feature correspondence constraints for label-free semantic guidance. We further introduce visible token joint encoding and video-domain augmentation invariance regularization. Our method achieves significant improvements over state-of-the-art methods across multiple video understanding benchmarks. Ablation studies confirm its enhanced capability in capturing action semantics and superior downstream transfer performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Current video-based Masked Autoencoders (MAEs) primarily focus on learning effective spatiotemporal representations from a visual perspective, which may lead the model to prioritize general spatial-temporal patterns but often overlook nuanced semantic attributes like specific interactions or sequences that define actions - such as action-specific features that align more closely with human cognition for space-time correspondence. This can limit the model's ability to capture the essence of certain actions that are contextually rich and continuous. Humans are capable of mapping visual concepts, object view invariance, and semantic attributes available in static instances to comprehend natural dynamic scenes or videos. Existing MAEs for videos and static images rely on separate datasets for videos and images, which may lack the rich semantic attributes necessary for fully understanding the learned concepts, especially when compared to using video and corresponding sampled frame images together. To this end, we propose CrossVideoMAE an end-to-end self-supervised cross-modal contrastive learning MAE that effectively learns both video-level and frame-level rich spatiotemporal representations and semantic attributes. Our method integrates mutual spatiotemporal information from videos with spatial information from sampled frames within a feature-invariant space, while encouraging invariance to augmentations within the video domain. This objective is achieved through jointly embedding features of visible tokens and combining feature correspondence within and across modalities, which is critical for acquiring rich, label-free guiding signals from both video and frame image modalities in a self-supervised manner. Extensive experiments demonstrate that our approach surpasses previous state-of-the-art methods and ablation studies validate the effectiveness of our approach.

Problem

• Enhances video and image semantic understanding.
• Integrates spatiotemporal and spatial information effectively.
• Improves self-supervised cross-modal representation learning.

Innovation

• Self-supervised cross-modal contrastive learning
• Joint embedding of video and frame features
• Feature-invariant space for spatiotemporal integration