Fast Byte Latent Transformer

2026-05-08

AI Summary

This work addresses the impractical inference efficiency of byte-level language models, which stems from their autoregressive, byte-by-byte generation process. To overcome this limitation, the authors introduce the BLT family of methods—BLT-D, BLT-S, and BLT-DV—that jointly leverage block-level diffusion training objectives, parallel byte decoding, and self-speculative generation mechanisms. This approach enables, for the first time, highly efficient parallel inference in byte-level models. The proposed techniques substantially reduce the number of forward passes and memory bandwidth consumption—by over 50%—while preserving high generation quality. Consequently, the method achieves a significant acceleration in inference speed, effectively breaking the practicality bottleneck that has long hindered the deployment of byte-level language models.

Abstract

Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte autoregressive generation. We address this bottleneck in the Byte Latent Transformer (BLT) through new training and generation techniques. First, we introduce BLT Diffusion (BLT-D), a new model and our fastest BLT variant, trained with an auxiliary block-wise diffusion objective alongside the standard next-byte prediction loss. This enables an inference procedure that generates multiple bytes in parallel per decoding step, substantially reducing the number of forward passes required to generate a sequence. Second, we propose two extensions inspired by speculative decoding that trade some of this speed for higher generation quality: BLT Self-speculation (BLT-S), in which BLT's local decoder continues generating past its normal patch boundaries to draft bytes, which are then verified with a single full-model forward pass; and BLT Diffusion+Verification (BLT-DV), which augments BLT-D with an autoregressive verification step after diffusion-based generation. All methods may achieve an estimated memory-bandwidth cost over 50% lower than BLT on generation tasks. Each approach offers its own unique advantages, together removing key barriers to the practical use of byte-level LMs.

Problem

• byte-level language models
• autoregressive generation
• generation speed
• inference efficiency
• sequence generation

Innovation

• Byte-level Language Models
• Parallel Decoding
• Diffusion-based Generation
• Speculative Decoding
• Latent Transformer