🤖 AI Summary
Existing adversarial contrastive learning (ACL) methods improve robustness under linear probing but lack a theoretical analysis linking unsupervised contrastive loss to robust supervised performance.
Method: This paper establishes, for the first time, a theoretical bridge between these two objectives by deriving generalization bounds and decomposing robust error, thereby identifying the loss components that dominantly govern robustness. Guided by this analysis, we propose a novel self-supervised pretraining objective within the contrastive learning framework.
Contribution/Results: We theoretically prove that penalizing a specific loss term significantly reduces robust error. Empirically, our proposed loss variant improves robust accuracy by 2.3–4.1% on benchmarks including ImageNet-C. This work provides both an interpretable theoretical foundation and a practical optimization pathway for robust self-supervised learning.
📝 Abstract
Contrastive learning (CL) is a self-supervised training paradigm that allows us to extract meaningful features without any label information. A typical CL framework is divided into two phases, where it first tries to learn the features from unlabelled data, and then uses those features to train a linear classifier with the labeled data. While a fair amount of existing theoretical works have analyzed how the unsupervised loss in the first phase can support the supervised loss in the second phase, none has examined the connection between the unsupervised loss and the robust supervised loss, which can shed light on how to construct an effective unsupervised loss for the first phase of CL. To fill this gap, our work develops rigorous theories to dissect and identify which components in the unsupervised loss can help improve the robust supervised loss and conduct proper experiments to verify our findings.
