Subobject-level Image Tokenization

2024-02-22

arXiv.org

AI Summary

Existing patch-based image tokenization methods neglect visual morphological structure, limiting models’ efficiency and effectiveness in image understanding. To address this, we propose a sub-object-level adaptive tokenization method that integrates lightweight boundary detection with Watershed segmentation, yielding the first human-cognition-aligned, pixel-complete tokenizer—EPOC—that supports both object- and part-level representations. EPOC synergizes superpixels, Segment Anything Model (SAM), and arbitrarily shaped token embedding to generate morphologically consistent and semantically unambiguous visual tokens. Experiments demonstrate that EPOC achieves segmentation accuracy approaching human annotation across five benchmark datasets. When integrated into vision-language models (VLMs), it accelerates training convergence, enhances generalization, and significantly reduces token count—by up to 60%—without compromising performance. This work establishes a new paradigm for structure-aware, cognitively grounded visual tokenization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Lexical Semantics and Morphology

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Patch-based image tokenization ignores the morphology of the visual world, limiting effective and efficient learning of image understanding. Inspired by subword tokenization, we introduce subobject-level adaptive token segmentation and explore several approaches, including superpixel, SAM, and a proposed Efficient and PanOptiC (EPOC) image tokenizer. Our EPOC combines boundary detection -- a simple task that can be handled well by a compact model -- with watershed segmentation, which inherently guarantees no pixels are left unsegmented. Intrinsic evaluations across 5 datasets demonstrate that EPOC's segmentation aligns well with human annotations of both object- and part-level visual morphology, producing more monosemantic tokens and offering substantial efficiency advantages. For extrinsic evaluation, we designed a token embedding that handles arbitrary-shaped tokens, and trained VLMs with different tokenizers on 4 datasets of object recognition and detailed captioning. The results reveal that subobject tokenization enables faster convergence and better generalization while using fewer visual tokens.

Problem

• Improves image tokenization by focusing on subobject-level segmentation.
• Introduces EPOC tokenizer combining boundary detection and watershed segmentation.
• Enhances visual language models with faster convergence and better generalization.

Innovation

• Subobject-level adaptive token segmentation introduced
• EPOC combines boundary detection with watershed segmentation
• Token embedding designed for arbitrary-shaped tokens