Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More

2025-02-06

AI Summary

This work investigates information loss induced by patchification in vision Transformers and its impact on visual understanding. To address this, we conduct systematic multi-scale patch experiments—including pixel-level (1×1) tokenization—alongside ultra-long sequence modeling (50,176 tokens) and cross-architecture (ViT/Mamba), cross-task, and cross-resolution evaluations. Our study establishes, for the first time, a universal scaling law governing the relationship between patch size and model performance. We empirically demonstrate that pixel-level tokenization is not only feasible but substantially improves accuracy—achieving 84.6% top-1 accuracy on ImageNet-1k with ViT-Base—while markedly reducing task-specific decoder reliance under small patches. These findings provide both theoretical insight into representational bottlenecks in visual compression and empirical support for designing lossless tokenization paradigms.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Large-scale security measurements

Abstract

Since the introduction of Vision Transformer (ViT), patchification has long been regarded as a de facto image tokenization approach for plain visual architectures. By compressing the spatial size of images, this approach can effectively shorten the token sequence and reduce the computational cost of ViT-like plain architectures. In this work, we aim to thoroughly examine the information loss caused by this patchification-based compressive encoding paradigm and how it affects visual understanding. We conduct extensive patch size scaling experiments and excitedly observe an intriguing scaling law in patchification: the models can consistently benefit from decreased patch sizes and attain improved predictive performance, until it reaches the minimum patch size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse architectures such as ViT and the recent Mamba models. Moreover, as a by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the visual sequence to an exceptional length of 50,176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models. Code is available at https://github.com/wangf3014/Patch_Scaling.

Problem

• Analyzes information loss in patchification-based encoding.
• Explores scaling laws for patch sizes in visual models.
• Assesses impact of patch size on model performance.

Innovation

• Patchification reduces computational cost
• Smaller patch sizes enhance performance
• Scaling up to 50,176 tokens achieved