The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training

2026-06-28

AI Summary

This work addresses the vulnerability of self-supervised pre-trained models to backdoor attacks in black-box settings, where existing defenses—relying on labels, attack patterns, or training data—struggle to generalize. The authors propose a novel test-time defense framework that is attack-agnostic, model-agnostic, and modality-agnostic. They formalize the Platonic representation hypothesis as a conditional energy function, enabling backdoor detection and purification without any prior knowledge. Detection is trained via noise contrastive estimation, while purification leverages denoising score matching and multiple independent pre-trained encoders to construct reference representations. Theoretical analysis establishes a lower bound linking the energy gap to mutual information. Extensive experiments demonstrate that the method significantly outperforms existing approaches across diverse self-supervised encoders and over ten backdoor attack variants, exhibiting strong generalization capabilities.

Abstract

Self-supervised learning (SSL) pretrained models have become a dominant paradigm for visual representation learning, but they are vulnerable to backdoor attacks. Existing defenses struggle to defend against such attacks in a fully black-box setting because they often require access to labels, attack patterns, or training data. To tackle this issue, we propose a new attack-agnostic, model-agnostic, and modality-agnostic black-box test-time defense paradigm, called \emph{Platonic Representation Defense}. It is inspired by the Platonic Representation Hypothesis, which suggests that large-scale independently trained encoders converge toward compatible projections of the same underlying reality. We formalize this idea as a conditional energy function defined over source representations and a set of reference representations. The energy function is trained for detection through noise-contrastive estimation and for representation purification through denoising score matching. Theoretically, the energy gap between matched and mismatched samples is lower bounded by the mutual information between source and reference representations. We demonstrate the effectiveness of our method on multiple self-supervised encoders and more than 10 attacks. The method can perform both representation detection and purification, and achieves substantial performance gains across multiple attacks. Code is available \href{https://github.com/jsrdcht/Platonic-Representation-Defense}{here}.

Problem

• backdoor defense
• self-supervised learning
• black-box setting
• pretrained models
• visual representation learning

Innovation

• Platonic Representation Defense
• backdoor defense
• self-supervised learning
• black-box defense
• representation purification