A Controlled Study of Self-Supervised Image and Video Pretraining under Limited Resources

2026-08-13

AI Summary

This study systematically evaluates the effectiveness of self-supervised learning objectives—including contrastive learning, masked autoencoding, and diffusion models—for image and video pretraining under constrained data and computational budgets. By unifying datasets, architectures, and compute resources, the work demonstrates for the first time that DINOv2 achieves consistently superior performance in resource-limited settings. Furthermore, it reveals that joint image–video pretraining substantially enhances performance on semantic tasks such as classification and segmentation, yet simultaneously degrades results on geometric perception tasks like tracking and camera pose estimation. These findings highlight an inherent trade-off between learning semantic and geometric representations in self-supervised vision systems.

Abstract

Visual foundation models are a cornerstone of image and video understanding but typically require large amounts of data and computation. The current scale required for pretraining visual foundation models may be unsustainable or unnecessary, and significant benefits arise when effective models can be obtained with fewer resources. To better understand how self-supervised learning (SSL) objectives behave under resource constraints, we conduct a controlled study of image and video SSL objectives under matched data, architecture, and compute budgets. We compare contrastive, reconstruction, feature-prediction, and diffusion objectives and evaluate both standalone and jointly trained image-video SSL formulations across a diverse set of image and video understanding tasks. Our results show that DINOv2-style pretraining consistently provides the strongest overall performance under limited resources. Furthermore, combining DINOv2 with video SSL objectives such as VideoMAE substantially improves image classification and segmentation performance, but degrades video tracking and camera-pose estimation performance, revealing an important tradeoff between semantic and geometric representation learning. These findings suggest that combining image and video SSL objectives can be beneficial in resource-limited settings, while highlighting the need for improved methods that better balance semantic, temporal, and geometric supervision.

Problem

• self-supervised learning
• visual foundation models
• limited resources
• image and video pretraining
• representation learning

Innovation

• self-supervised learning
• resource-constrained pretraining
• image-video foundation models
• DINOv2
• representation tradeoff