Exploring Structural Degradation in Dense Representations for Self-supervised Learning

2025-10-20

AI Summary

This paper identifies a counterintuitive phenomenon in self-supervised learning (SSL): prolonged pretraining degrades performance on dense prediction tasks—e.g., semantic segmentation—a behavior termed *Self-supervised Dense Degradation* (SDD). Addressing the fundamental challenge of evaluating dense representation quality without labels, we propose the first *Dense Representation Structure Estimator* (DSE), which jointly quantifies class-wise correlation and effective dimensionality; we theoretically and empirically establish its strong correlation with downstream task performance. Leveraging DSE, we further devise unsupervised model selection and regularization strategies. Extensive experiments across four benchmarks and sixteen SSL methods demonstrate that DSE-guided model selection improves mean Intersection-over-Union (mIoU) by 3.0% on average, while DSE-based regularization consistently mitigates SDD with negligible computational overhead.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this work, we observe a counterintuitive phenomenon in self-supervised learning (SSL): longer training may impair the performance of dense prediction tasks (e.g., semantic segmentation). We refer to this phenomenon as Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across sixteen state-of-the-art SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, measuring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge. To tackle this issue, we introduce a Dense representation Structure Estimator (DSE), composed of a class-relevance measure and an effective dimensionality measure. The proposed DSE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and a DSE-based regularization method. Experiments on sixteen SSL methods across four benchmarks confirm that model selection improves mIoU by $3.0%$ on average with negligible computational cost. Additionally, DSE regularization consistently mitigates the effects of dense degradation. Code is available at https://github.com/EldercatSAM/SSL-Degradation.

Problem

• Longer SSL training impairs dense prediction task performance
• Evaluating dense performance without annotations remains challenging
• Proposing metric and methods to mitigate structural degradation

Innovation

• Proposes Dense Structure Estimator for SSL
• Introduces model selection strategy using DSE
• Develops DSE-based regularization against degradation