Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representations

2026-02-01

AI Summary

This work addresses the limitation of existing Joint Embedding Predictive Architecture (JEPA) models, which typically learn dense representations and struggle to effectively capture the sparsity essential for efficient representation learning. The authors propose RDMReg, a novel regularization method that, for the first time within the JEPA framework, unifies tunable sparsity with the maximum entropy principle. By modeling latent representations with a modified generalized Gaussian distribution, incorporating a sliced two-sample distribution matching loss, and combining ℓ₀/ℓₚ-norm constraints with non-negative sparse representation learning, RDMReg explicitly controls the sparse structure of the learned representations. This approach rigorously generalizes prior Gaussian-assumption-based methods and achieves competitive downstream performance on image classification benchmarks while yielding sparse, non-negative, and information-rich representations, effectively balancing sparsity and task performance.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Joint-Embedding Predictive Architectures (JEPA) learn view-invariant representations and admit projection-based distribution matching for collapse prevention. Existing approaches regularize representations towards isotropic Gaussian distributions, but inherently favor dense representations and fail to capture the key property of sparsity observed in efficient representations. We introduce Rectified Distribution Matching Regularization (RDMReg), a sliced two-sample distribution-matching loss that aligns representations to a Rectified Generalized Gaussian (RGG) distribution. RGG enables explicit control over expected $\ell_0$ norm through rectification, while preserving maximum-entropy up to rescaling under expected $\ell_p$ norm constraints. Equipping JEPAs with RDMReg yields Rectified LpJEPA, which strictly generalizes prior Gaussian-based JEPAs. Empirically, Rectified LpJEPA learns sparse, non-negative representations with favorable sparsity-performance trade-offs and competitive downstream performance on image classification benchmarks, demonstrating that RDMReg effectively enforces sparsity while preserving task-relevant information.

Problem

• sparsity
• joint-embedding predictive architectures
• representation learning
• distribution matching
• rectified generalized gaussian

Innovation

• Rectified LpJEPA
• Joint-Embedding Predictive Architecture
• sparsity
• maximum-entropy representation
• distribution matching regularization