Simplifying DINO via Coding Rate Regularization

2025-02-14

AI Summary

DINO and DINOv2 achieve strong performance in unsupervised visual representation learning but suffer from training complexity, susceptibility to representational collapse, and high sensitivity to hyperparameters—severely limiting generalization and adaptability. To address these issues, we propose SimDINO and SimDINOv2, the first methods to explicitly incorporate *rate minimization* as a core collapse-mitigation mechanism. By replacing redundant components—including momentum teachers, multi-crop augmentation, and intricate learning-rate scheduling—with a lightweight, explicit regularization term, our framework substantially simplifies self-distillation. This design markedly improves training stability and convergence speed while reducing architectural and hyperparameter sensitivity. Extensive experiments demonstrate Pareto-optimal improvements across downstream tasks, including image classification and semantic segmentation. SimDINO/SimDINOv2 establishes a new paradigm for self-supervised pretraining that is lightweight, robust, and deployment-friendly.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

DINO and DINOv2 are two model families being widely used to learn representations from unlabeled imagery data at large scales. Their learned representations often enable state-of-the-art performance for downstream tasks, such as image classification and segmentation. However, they employ many empirically motivated design choices and their training pipelines are highly complex and unstable -- many hyperparameters need to be carefully tuned to ensure that the representations do not collapse -- which poses considerable difficulty to improving them or adapting them to new domains. In this work, we posit that we can remove most such-motivated idiosyncrasies in the pre-training pipelines, and only need to add an explicit coding rate term in the loss function to avoid collapse of the representations. As a result, we obtain highly simplified variants of the DINO and DINOv2 which we call SimDINO and SimDINOv2, respectively. Remarkably, these simplified models are more robust to different design choices, such as network architecture and hyperparameters, and they learn even higher-quality representations, measured by performance on downstream tasks, offering a Pareto improvement over the corresponding DINO and DINOv2 models. This work highlights the potential of using simplifying design principles to improve the empirical practice of deep learning.

Problem

• Simplify DINO training pipeline
• Avoid representation collapse
• Enhance model robustness

Innovation

• Simplifies DINO via coding rate
• Enhances robustness to design choices
• Improves representation quality significantly