Learning from Semantic Dictionaries: Discriminative Codebook Contrastive Learning for Unified Visual Representation and Generation

2026-05-24

AI Summary

This work addresses the challenge of unifying discriminative and generative visual models, which often suffer from semantic inconsistency. To this end, the authors propose the LEASE framework, which operates in a discrete token space and jointly optimizes masked token reconstruction and a contrastive loss with adaptive centroid weighting through paired generative-discriminative codebooks. Notably, LEASE establishes a unified latent space that simultaneously supports high-quality generation and strong representation learning—without relying on data augmentation, teacher models, or online tokenizers. On ImageNet-1K, the method achieves a 1.7% improvement in linear probing accuracy, reduces unconditional generation FID by 1.26, increases Inception Score by 10.19, and consistently outperforms prior approaches such as MAGE and Sorcen in few-shot learning, transfer performance, and robustness.

Abstract

Discriminative and generative vision models excel in their respective domains but remain semantically misaligned, hindering progress toward unified visual learning. We introduce LEASE (LEArning from SEmantic Dictionaries), a self-supervised framework that bridges this gap using a paired generative-discriminative codebook design. LEASE operates entirely in a discrete token space produced through a one-time precomputation step, enabling efficient training without data augmentations, teacher models, or online tokenizers. LEASE integrates two complementary objectives: a masked token reconstruction loss that captures fine-grained generative detail, and a codebook contrast loss that aligns encoder features with discriminative semantics via adaptive centroid weighting. This dual supervision yields a unified latent space that supports both high-quality generation and strong representation learning. On ImageNet-1K, LEASE achieves state-of-the-art unified performance, outperforming prior VQGAN-based methods such as MAGE and Sorcen across linear probing (up to +1.7%), unconditional generation (-1.26 FID and +10.19 IS w.r.t MAGE), few-shot learning (+0.56% on average against Sorcen), transfer (+0.75% average improvement against MAGE and Sorcen), and robustness benchmarks (+5.86% and +4.25% average improvement against MAGE and Sorcen, respectively). It also competes favorably with domain-specialized contrastive and generative models while surpassing previous MIM methods. The unsupervised LEASE model can also be extended to conditional generation by building upon its learned representations, proving competitive with specialized baselines. Overall, LEASE provides an efficient and effective step toward general-purpose vision models that jointly understand and generate visual content.

Problem

• unified visual learning
• semantic alignment
• discriminative models
• generative models
• visual representation

Innovation

• codebook contrastive learning
• unified visual representation
• discrete token space
• masked token reconstruction
• self-supervised vision model