SpeLLM: Character-Level Multi-Head Decoding

2025-07-22

AI Summary

Expanding the vocabulary of large language models (LLMs) incurs substantial computational overhead in the output projection layer’s linear transformation, becoming a critical bottleneck. To address this, we propose a multi-head character-level decoding architecture that decouples input and output vocabularies: multiple independent linear heads predict character sequences in parallel, and a self-distillation-based conversion mechanism enables effective knowledge transfer from the original word-level model. Our approach requires no modifications to the pre-trained backbone and is plug-and-play compatible with mainstream LLMs. Experiments across four state-of-the-art LLMs demonstrate that downstream task performance remains on par with baselines, while average inference latency decreases by 5.1%. Moreover, the method significantly enhances support for low-resource languages and improves vocabulary expansion efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Scaling LLM vocabulary is often used to reduce input sequence length and alleviate attention's quadratic cost. Yet, current LLM architectures impose a critical bottleneck to this procedure: the output projection layer scales linearly with vocabulary size, rendering substantial expansion impractical. We propose SpeLLM, a method that decouples input and output vocabularies by predicting character-level strings through multiple output heads. In SpeLLM, each of the $k$ linear heads predicts a single character simultaneously, enabling the model to represent a much larger output space using smaller, independent linear heads. We present a self-distillation approach for converting a standard LLM to a SpeLLM. Our experiments with four pre-trained LLMs show their SpeLLM variants achieve competitive performance on downstream tasks while reducing runtime by 5.1% on average across models. Our approach provides a potential avenue for reducing LLM costs, while increasing support for underrepresented languages and domains.

Problem

• Reducing LLM output projection layer's vocabulary size bottleneck
• Decoupling input and output vocabularies via multi-head character prediction
• Enabling larger output space with smaller independent linear heads

Innovation

• Decouples input and output vocabularies using multi-head decoding
• Predicts character-level strings with multiple linear heads
• Self-distillation converts standard LLM to SpeLLM efficiently