Towards Semantic Equivalence of Tokenization in Multimodal LLM

2024-06-07

arXiv.org

AI Summary

Existing multimodal large language models (MLLMs) employ fixed-grid visual tokenization, which often disrupts image semantic integrity and causes cross-modal representation mismatch. To address this, we propose SeTok—a dynamic, semantic-equivalent visual tokenizer—that introduces a novel semantic-driven, adaptive clustering tokenization paradigm: it replaces rigid grids with an end-to-end differentiable encoder to produce semantically coherent visual units; and supports complexity-aware token count generation, balancing structural fidelity and fine-grained detail. Our method integrates dynamic hierarchical clustering, semantic feature grouping, and cross-modal alignment optimization. Evaluated across 12 vision-language understanding and generation benchmarks, SeTok achieves an average improvement of 3.2%, significantly enhancing fine-grained reasoning and long-range semantic consistency. The open-sourced SeTok model has been widely adopted by the research community.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in processing vision-language tasks. One of the crux of MLLMs lies in vision tokenization, which involves efficiently transforming input visual signals into feature representations that are most beneficial for LLMs. However, existing vision tokenizers, essential for semantic alignment between vision and language, remain problematic. Existing methods aggressively fragment visual input, corrupting the visual semantic integrity. To address this, this paper proposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok), which groups visual features into semantic units via a dynamic clustering algorithm, flexibly determining the number of tokens based on image complexity. The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features. The proposed MLLM (Setokim) equipped with SeTok significantly demonstrates superior performance across various tasks, as evidenced by our experimental results. The project page is at https://chocowu.github.io/SeTok-web/.

Problem

• Enhances semantic alignment in vision tokenization
• Preserves visual semantic integrity in MLLMs
• Improves feature representation for multimodal tasks

Innovation

• Dynamic Semantic-Equivalent Vision Tokenizer
• Groups visual features into units
• Flexibly determines token number