From Tokens to Words: On the Inner Lexicon of LLMs

2024-10-08

arXiv.org

AI Summary

This work investigates whether large language models (LLMs) implicitly construct word-level representations, despite being trained on subword tokenization. Method: We analyze intermediate-layer activations, apply controlled subword perturbations, re-inject cross-subword representations, and evaluate downstream task performance. Contribution/Results: We empirically demonstrate—for the first time—that LLMs internally maintain an implicit lexicon transcending their tokenizer: subwords are implicitly merged into full-word representations at late layers, exhibiting robustness to tokenization boundaries, spelling errors, and out-of-vocabulary words. Leveraging this insight, we propose a fine-tuning-free lexical expansion method that dynamically injects novel word representations. Experiments show substantial reductions in input length and decoding steps, lowering memory footprint and latency, with negligible accuracy degradation. Moreover, we validate the model’s intrinsic capacity for compositional semantics—generating meaningful representations for unseen word forms.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Natural language is composed of words, but modern large language models (LLMs) process sub-words as input. A natural question raised by this discrepancy is whether LLMs encode words internally, and if so how. We present evidence that LLMs engage in an intrinsic detokenization process, where sub-word sequences are combined into coherent whole-word representations at their last token. Our experiments show that this process primarily takes place within the early and middle layers of the model. We further demonstrate its robustness to arbitrary splits (e.g.,"cats"to"ca"and"ts"), typos, and importantly-to out-of-vocabulary words: when feeding the last token internal representations of such words to the model as input, it can"understand"them as the complete word despite never seeing such representations as input during training. Our findings suggest that LLMs maintain a latent vocabulary beyond the tokenizer's scope. These insights provide a practical, finetuning-free application for expanding the vocabulary of pre-trained models. By enabling the addition of new vocabulary words, we reduce input length and inference iterations, which reduces both space and model latency, with little to no loss in model accuracy.

Problem

• Investigates if LLMs internally encode words from sub-words.
• Explores how LLMs combine sub-words into coherent word representations.
• Demonstrates LLMs' ability to understand out-of-vocabulary words.

Innovation

• LLMs combine sub-words into whole-word representations
• Robust to arbitrary splits, typos, and out-of-vocabulary words
• Expands vocabulary without finetuning, reducing input length