Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training?

2026-07-14

AI Summary

This study systematically investigates the interplay between self-supervised and supervised learning objectives by comparing the pretrain-then-finetune (PFT) and joint training (JT) paradigms across varying label budgets, task types, and data domains. For the first time, it conducts a comprehensive evaluation of representation quality, robustness, and cross-domain generalization using eight representative self-supervised methods across diverse visual domains—including natural images, medical imaging, emergency response, and remote sensing. The findings reveal that JT is more efficient and robust under low-label regimes, whereas PFT demonstrates greater reliability in specialized, complex domains. These results provide empirical guidance for selecting training strategies in practical applications and establish a new benchmark for hybrid self-supervised semi-supervised learning.

Abstract

Self-supervision is a powerful technique for learning visual representations from unlabeled data. Existing techniques primarily adopt a two-stage approach for self-supervised learning (SSL): a pretraining stage on unlabeled data followed by a finetuning stage on labeled data. While this pipeline has demonstrated extreme effectiveness, the interaction between self-supervised and supervised learning objectives remains insufficiently understood. In this work, we systematically investigate whether jointly optimizing the self-supervised and supervised objectives during training provides a better alternative. We compare two training paradigms: (1) the aforementioned pretraining followed by finetuning (PFT) and (2) joint training (JT), where self-supervised and supervised losses are optimized simultaneously in the same network. Across eight representative SSL methods and diverse computer vision tasks on natural, medical, crisis response, and remote sensing data, we evaluate performance under varying percentages of labeled data. Our results reveal that the relative effectiveness of PFT and JT depends strongly on the task at hand, the availability of labeled data, and the complexity of the domain. We find that JT consistently improves data and training efficiency while being robust in low-label settings, while PFT is more reliable in more specialized domains. We further analyze representation quality, robustness, and cross-domain generalization, providing new insights into how self-supervised and supervised objectives interact during optimization. We establish a comprehensive empirical benchmark for hybrid SSL-based semi-supervised learning and offer practical guidance for selecting appropriate training strategies across diverse vision applications.

Problem

• Self-Supervised Learning
• Joint Training
• Pretrain-Finetuning
• Visual Representation Learning
• Semi-Supervised Learning

Innovation

• self-supervised learning
• joint training
• pretrain-finetune
• semi-supervised learning
• representation learning