MultiTok: Variable-Length Tokenization for Efficient LLMs Adapted from LZW Compression

2024-10-28

arXiv.org

AI Summary

To address high data redundancy, excessive computational overhead, and low training efficiency in large language model (LLM) training, this paper proposes MultiTok—a variable-length, multi-token tokenization method inspired by LZW lossless compression. Its core innovation lies in the first application of LZW’s dynamic dictionary construction to tokenization: MultiTok identifies frequent n-grams and merges them into composite tokens, thereby compressing sequence length while preserving semantic integrity. MultiTok seamlessly integrates with mainstream architectures (e.g., BERT, GPT-2), accelerates training by 2.5×, reduces input data volume by over 30%, and maintains baseline-level accuracy on standard NLU and language modeling benchmarks. This work introduces a novel, data-compression-driven tokenization paradigm for efficient and scalable LLM training.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models have drastically changed the prospects of AI by introducing technologies for more complex natural language processing. However, current methodologies to train such LLMs require extensive resources including but not limited to large amounts of data, expensive machinery, and lengthy training. To solve this problem, this paper proposes a new tokenization method inspired by universal Lempel-Ziv-Welch data compression that compresses repetitive phrases into multi-word tokens. With MultiTok as a new tokenizing tool, we show that language models are able to be trained notably more efficiently while offering a similar accuracy on more succinct and compressed training data. In fact, our results demonstrate that MultiTok achieves a comparable performance to the BERT and GPT-2 standards as both a stand-alone tokenizer and an add-on to existing tokenizers while also providing close to 2.5x faster training with more than 30% less training data.

Problem

• Large Language Models
• Computational Resources
• Scalability

Innovation

• MultiTok
• LZW Compression
• Efficient Training