Next-Embedding Prediction Makes Strong Vision Learners

2025-12-18

AI Summary

This work investigates the feasibility of generative pre-training for visual self-supervised learning. We propose Next-Embedding Predictive Autoregression (NEPA), a paradigm that abandons pixel reconstruction, discrete tokenization, and contrastive losses, instead performing causal autoregressive prediction directly over continuous patch embeddings in latent space. NEPA employs a Vision Transformer (ViT) backbone with causal masking and gradient truncation, enabling end-to-end, head-free modeling while supporting cross-modal extensibility. On ImageNet-1K fine-tuning, NEPA achieves 83.8% and 85.3% top-1 accuracy with ViT-Base and ViT-Large, respectively. In transfer learning to ADE20K semantic segmentation, it significantly outperforms leading self-supervised methods. Our key contribution is the first direct application of generative pre-training to continuous visual embedding prediction—establishing a novel, unified representation learning paradigm that bridges language-inspired autoregression and vision foundation models.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.

Problem

• Develops a generative pretraining method for vision models
• Shifts from learning representations to learning predictive models
• Explores embedding prediction as a self-supervised learning objective

Innovation

• Predict future patch embeddings using autoregression
• Use causal masking and stop gradient techniques
• Train Transformer with next-embedding prediction objective