KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learning

2025-12-22

AI Summary

To address the training instability and limited architectural flexibility of self-supervised Joint Embedding Predictive Architectures (JEPA) in Euclidean space, this paper proposes a novel kernel-based regularization framework. Our method formulates latent-space distribution matching in JEPA as kernelized Maximum Mean Discrepancy (MMD), thereby establishing the first scalable family of kernelized JEPAs. We derive the closed-form asymptotic limit of high-dimensional sliced MMD, enabling rigorous theoretical analysis for arbitrary kernels combined with isotropic Gaussian priors. Compared to conventional Epps–Pulley regularization, our approach significantly improves training stability and downstream generalization performance. Crucially, we provide formal theoretical guarantees demonstrating its strict superiority in terms of both convergence behavior and statistical consistency. The framework preserves JEPA’s predictive structure while enhancing robustness and expressivity through principled kernel embedding, offering a unified foundation for stable, theoretically grounded representation learning.

Technology Category

• Machine Learning: Kernel Methods
• Search and Optimization: Learning to Search
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent breakthroughs in self-supervised Joint-Embedding Predictive Architectures (JEPAs) have established that regularizing Euclidean representations toward isotropic Gaussian priors yields provable gains in training stability and downstream generalization. We introduce a new, flexible family of KerJEPAs, self-supervised learning algorithms with kernel-based regularizers. One instance of this family corresponds to the recently-introduced LeJEPA Epps-Pulley regularizer which approximates a sliced maximum mean discrepancy (MMD) with a Gaussian prior and Gaussian kernel. By expanding the class of viable kernels and priors and computing the closed-form high-dimensional limit of sliced MMDs, we develop alternative KerJEPAs with a number of favorable properties including improved training stability and design flexibility.

Problem

• Introduces KerJEPAs with kernel-based regularizers for self-supervised learning
• Expands kernel and prior options to compute sliced MMD limits
• Aims to improve training stability and design flexibility in models

Innovation

• Kernel-based regularizers for self-supervised learning
• Expanding kernel and prior classes for flexibility
• Closed-form high-dimensional limit of sliced MMDs