Tokenize Image as a Set

2025-03-20

AI Summary

This work addresses the limitations of conventional image generation frameworks, which rely on fixed-position, uniform-compression-ratio serialized latent representations. To overcome these constraints, we propose a dynamic image representation based on unordered token sets. Our method introduces three key innovations: (1) a semantic-aware dynamic tokenization mechanism that adaptively allocates encoding capacity according to local complexity; (2) Fixed-Sum Discrete Diffusion—a novel diffusion model jointly supporting discrete-valued tokens, fixed-length sequences, and sum-invariance; and (3) a bijective set transformation enabling reversible mapping between unordered token sets and discrete latent spaces. Experiments demonstrate substantial improvements in global contextual modeling and robustness to local perturbations, with superior generative quality compared to state-of-the-art serialized tokenization schemes. The code and pretrained models are publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This paper proposes a fundamentally new paradigm for image generation through set-based tokenization and distribution modeling. Unlike conventional methods that serialize images into fixed-position latent codes with a uniform compression ratio, we introduce an unordered token set representation to dynamically allocate coding capacity based on regional semantic complexity. This TokenSet enhances global context aggregation and improves robustness against local perturbations. To address the critical challenge of modeling discrete sets, we devise a dual transformation mechanism that bijectively converts sets into fixed-length integer sequences with summation constraints. Further, we propose Fixed-Sum Discrete Diffusion--the first framework to simultaneously handle discrete values, fixed sequence length, and summation invariance--enabling effective set distribution modeling. Experiments demonstrate our method's superiority in semantic-aware representation and generation quality. Our innovations, spanning novel representation and modeling strategies, advance visual generation beyond traditional sequential token paradigms. Our code and models are publicly available at https://github.com/Gengzigang/TokenSet.

Problem

• Introduces set-based tokenization for dynamic image representation.
• Develops dual transformation for modeling discrete sets effectively.
• Proposes Fixed-Sum Discrete Diffusion for enhanced image generation.

Innovation

• Set-based tokenization for image generation
• Dual transformation mechanism for set modeling
• Fixed-Sum Discrete Diffusion framework