ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

2026-03-03

AI Summary

This work proposes ByteFlow Net, an end-to-end hierarchical architecture that operates directly on raw bytes without relying on predefined subword tokenizers. Addressing the rigidity of fixed semantic granularity and limited generalization in conventional language models, ByteFlow Net employs a compression-driven dynamic segmentation mechanism to adaptively learn meaningful semantic units. It further introduces an encoding-rate-guided Top-K sparse selection strategy, enabling differentiable and static-graph-compatible adaptive chunking. Evaluated across multiple benchmarks, the model significantly outperforms both BPE-based Transformers and existing byte-level approaches, demonstrating the effectiveness and superiority of tokenizer-free language modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce \textbf{ByteFlow Net}, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries \emph{while preserving a static computation graph via Top-$K$ selection}. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.

Problem

• tokenization
• language modeling
• byte-level representation
• adaptive segmentation
• tokenizer-free

Innovation

• ByteFlow Net
• tokenizer-free
• adaptive segmentation
• compression-driven chunking
• byte-level modeling