Self-Supervised Learning with a Multi-Task Latent Space Objective

2026-02-05

AI Summary

This work addresses the training instability commonly induced by multi-crop strategies in predictor-based self-supervised learning. To mitigate this issue, the authors propose a multi-task asymmetric siamese network that assigns dedicated predictors to global, local, and masked crop views, treating each spatial transformation as an independent alignment task. The framework further incorporates Cutout-based masking augmentation to enhance representation consistency. Compatible with both ResNet and Vision Transformer (ViT) backbones, the proposed method achieves substantial improvements in representation learning performance on ImageNet, while maintaining strong training stability and broad applicability across architectures.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Self-supervised learning (SSL) methods based on Siamese networks learn visual representations by aligning different views of the same image. The multi-crop strategy, which incorporates small local crops to global ones, enhances many SSL frameworks but causes instability in predictor-based architectures such as BYOL, SimSiam, and MoCo v3. We trace this failure to the shared predictor used across all views and demonstrate that assigning a separate predictor to each view type stabilizes multi-crop training, resulting in significant performance gains. Extending this idea, we treat each spatial transformation as a distinct alignment task and add cutout views, where part of the image is masked before encoding. This yields a simple multi-task formulation of asymmetric Siamese SSL that combines global, local, and masked views into a single framework. The approach is stable, generally applicable across backbones, and consistently improves the performance of ResNet and ViT models on ImageNet.

Problem

• Self-Supervised Learning
• Multi-Crop Strategy
• Predictor Instability
• Siamese Networks
• Visual Representation Learning

Innovation

• Self-Supervised Learning
• Multi-Task Learning
• Siamese Networks
• Multi-Crop Strategy
• View-Specific Predictors