Vision Foundation Models as Generalist Tokenizers for Image Generation

2026-05-18

AI Summary

This work aims to construct an efficient and general-purpose image tokenizer to enhance both semantic fidelity and inference efficiency in generative models. Building upon a frozen vision foundation model (VFM), the authors propose a region-adaptive quantization framework to eliminate spatial redundancy and introduce a semantic reconstruction objective to ensure consistency between discrete and continuous latent representations. The study further reveals, for the first time, the critical influence of VFM pretraining objectives on tokenization efficacy. The proposed method achieves state-of-the-art results on ImageNet, attaining a discrete-generation gFID of 1.36 and a continuous-generation gFID of 1.25, enabling high-fidelity class-conditional synthesis without classifier-free guidance while accelerating training convergence by a factor of three.

Abstract

In this work, we explore the largely unexplored direction of building a generalist image tokenizer directly on top of a frozen vision foundation model (VFM). To build this tokenizer, we utilize a frozen VFM as the encoder and introduce two key innovations: (1) a region-adaptive quantization framework to eliminate spatial redundancy in standard 2D grid features, and (2) a semantic reconstruction objective that aligns the decoded outputs with the VFM's representations to preserve semantic fidelity. Grounded in these designs, we propose VFMTok, a generalist visual tokenizer capable of operating seamlessly in both discrete and continuous latent spaces. VFMTok achieves substantial improvements in synthesis quality while drastically enhancing token efficiency. For discrete autoregressive (AR) generation, it accelerates model convergence by \textbf{3 times} and achieves a state-of-the-art gFID of \textbf{1.36} on ImageNet class-conditional synthesis. Similarly, for continuous-space generation, integrating VFMTok with a denoising model yields an exceptional gFID of \textbf{1.25}. Furthermore, because the latent space inherently captures rich spatial semantics, VFMTok enables high-fidelity class-conditional synthesis without classifier-free guidance (\textbf{w/o CFG}) across both generative paradigms, significantly accelerating inference speed. Beyond these remarkable empirical results, we systematically investigate the underlying mechanisms of our approach. We discover that the specific self-supervised learning objectives utilized during VFM pre-training dictate its effectiveness as a tokenizer. Specifically, a VFM jointly optimized with global contrastive learning and latent masked image modeling provides the optimal representations for image tokenization. These insights establish a strong foundation and offer valuable guidance for the design of future image tokenizers.

Problem

• image tokenizer
• vision foundation model
• image generation
• semantic fidelity
• token efficiency

Innovation

• vision foundation model
• generalist tokenizer
• region-adaptive quantization
• semantic reconstruction
• classifier-free guidance