Three Necessary Principles for Self-Supervised Visual Representation Learning

2026-08-08

AI Summary

This work addresses the lack of a unified theoretical foundation in unsupervised visual representation learning, where existing methods struggle to simultaneously achieve semantic invariance, spatial structure modeling, and non-degenerate solutions. The authors propose three essential principles—observation, prediction, and regularization—and formulate them within a unified energy-based decomposition framework, offering the first formalization of core self-supervised learning criteria. Through rigorous analysis of gradient complementarity, convergence guarantees for momentum encoders, and a negative-sample-free alignment theory, the study exposes fundamental limitations of contrastive learning and momentum mechanisms, demonstrating that all three principles are indispensable. Controlled experiments, including block retrieval evaluations, confirm that optimal performance is attained only when these principles operate in concert.

Abstract

We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views, patch-level spatial prediction, and representational non-degeneracy. We formalize these as the observation, prediction, and regularization principles and prove (i) that combining observation and prediction without regularization admits the constant encoder as a global minimizer under negative-free alignment; (ii) that the two objectives are gradient-complementary and structurally non-conflicting at the encoder output; and (iii) that the momentum encoder converges to the same fixed point as the online encoder and provides no collapse guarantee at convergence. Contrastive alignment provides only self-limiting collapse resistance, formalized via an explicit gradient-decay argument. Dropping prediction withholds the spatial training signal by construction; dropping observation forfeits cross-view semantic invariance by construction; at the scale we study, no pair substitutes for the third. Every major self-supervised method is a special case of a single unified energy decomposition. We pair every theoretical claim with a controlled experiment, including a patch-retrieval evaluation for the spatial consequence of prediction.

Problem

• self-supervised learning
• visual representation
• semantic invariance
• spatial prediction
• non-degeneracy

Innovation

• self-supervised learning
• representation collapse
• semantic invariance
• spatial prediction
• non-degeneracy