SiamJEPA: On the Role of Siamese Student Encoders in JEPA

2026-07-04

AI Summary

This work addresses the limitations of existing Joint-Embedding Predictive Architectures (JEPAs), which typically employ a single encoder and consequently suffer from suboptimal representation learning efficiency and poor disentanglement. We propose SiamJEPA, a novel framework that introduces a masked Siamese student encoder paired with an exponential moving average (EMA) teacher network, enabling self-supervised learning through the prediction of latent embeddings of masked image regions. We demonstrate that the Siamese architecture is not merely a design choice but constitutes a crucial inductive bias that substantially enhances representational disentanglement and accelerates convergence during early training stages. Under linear probing on ImageNet with limited training budgets, SiamJEPA outperforms single-encoder JEPA variants and achieves higher accuracy than Masked Autoencoders (MAE), despite requiring significantly less training time.

Abstract

Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.

Problem

• Joint Embedding Predictive Architecture
• Siamese encoders
• self-supervised representation learning
• inductive bias
• predictive learning

Innovation

• Siamese encoders
• JEPA
• self-supervised learning
• representation learning
• inductive bias