LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives

2026-07-01

AI Summary

This work addresses the limitations of existing vision-language pretraining approaches, which predominantly rely on contrastive learning and struggle to produce high-quality visual features suitable for dense prediction tasks. The authors propose the first end-to-end, fully non-contrastive pretraining method that eliminates the need for negative samples, temperature scaling, momentum encoders, or teacher-student mechanisms. Their approach achieves stable large-scale semantic alignment through cross-modal target prediction (with gradient stopping), intra-modal distribution regularization, and joint training. Under a frozen backbone setting, the method achieves state-of-the-art performance on GQA, VQAv2, and POPE, while significantly outperforming contrastive baselines on dense prediction tasks such as semantic segmentation, all without compromising global semantic understanding.

Abstract

Vision-language pretraining remains dominated by contrastive objectives, whereas vision-only self-supervised learning has largely adopted non-contrastive methods. At the same time, the role of vision-language encoders has shifted: they are increasingly deployed not as zero-shot classifiers but as the frozen visual backbone of vision-language models and dense prediction systems, which consume the full grid of patch tokens rather than a single pooled embedding. We introduce LeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method. LeVLJEPA learns through cross-modal prediction with stop-gradient targets and per-modality distributional regularization, without negatives, temperature, momentum encoder, or teacher-student schedule, and trains stably at large scale. We find that the resulting encoder provides markedly stronger dense semantic features for downstream use: as a frozen vision-language-model backbone, LeVLJEPA is the strongest of the evaluated encoders across GQA, VQAv2, and POPE under two distinct language models, and outperforms contrastive baselines on semantic segmentation, while remaining on par on global readouts such as linear probing. These results establish non-contrastive pretraining as an effective means of producing dense semantic vision features.

Problem

• vision-language pretraining
• non-contrastive learning
• dense prediction
• frozen backbone
• semantic features

Innovation

• non-contrastive learning
• vision-language pretraining
• dense semantic features
• cross-modal prediction
• frozen backbone