🤖 AI Summary
This work addresses the challenge that existing language models, constrained by model-specific tokenizers, struggle to efficiently transfer knowledge or flexibly adjust segmentation boundaries across architectures. The authors propose decoupling the “next-byte prediction” and “segment boundary delineation” distributions within byte-level language models, leveraging their shared output space to enable precise, alignment-free knowledge transfer. Through distributional decoupling analysis, cross-model capability transfer experiments, and boundary behavior measurements, the study provides the first systematic evidence that modeling capacity and boundary decisions can be approximately controlled independently. These findings offer preliminary support for the decoupling hypothesis and lay the groundwork for a universal model ecosystem standardized on byte-level interfaces, potentially enabling low-cost, routine model capability transfer and boundary reconfiguration.
📝 Abstract
Byte-level language models are usually argued for on the grounds of robustness, multilingual fairness, and character-level skills. We point to a different, structural advantage: because they read and write bytes, any two of them share an output space, so knowledge transfer between them is exact and independent of how either was originally tokenized. We hypothesize that the two distributions a byte-level model produces, one over the next byte, one over where its patch boundaries fall, can be disentangled and changed almost independently. A model could absorb a teacher's capability while keeping its own boundaries, or change how it places those boundaries while keeping its capabilities. We lay out the two experiments that would settle the hypothesis, alongside preliminary measurements of the properties they rest on. We argue that the community should move toward a byte-level interface as a shared standard: if the hypothesis holds, then once byte-level models are the norm, transferring capabilities and reshaping boundaries between them become cheap and routine, free of the per-model tokenizer that blocks them today.