Sumi: Open Uniform Diffusion Language Model from Scratch

2026-06-17

AI Summary

This work addresses the absence of large-scale, from-scratch pretrained unified diffusion language models (UDLMs) in existing literature, which has hindered systematic investigation into their scaling behavior and controllable generation mechanisms. We present Sumi, the first open-source 7B-parameter UDLM trained entirely from scratch on a 1.5T-token education-oriented data mixture. Sumi leverages a unified diffusion framework that supports updating arbitrary tokens at any diffusion step and is trained using efficient distributed strategies. Empirical results show that Sumi matches the performance of similarly sized autoregressive models on knowledge, reasoning, and programming tasks, though it lags slightly on commonsense benchmarks. To foster further research, we publicly release the complete training recipe, data mixture, model weights, and checkpoints, thereby filling a critical gap in the field.

Abstract

Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion language models (UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and large token budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior, generation dynamics, controllability, and trade-offs against established autoregressive and masked diffusion models. To this end, we introduce Sumi ("ink" in Japanese), a fully open 7B uniform diffusion language model pretrained from scratch on 1.5T tokens. Sumi performs competitively with autoregressive models trained at comparable token budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavy data mixture is a likely contributor. We release our model weights, checkpoints, and full training recipe, including a complete specification of the data mixture over publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.

Problem

• uniform diffusion language model
• pretraining from scratch
• large-scale language modeling
• diffusion models
• scaling behavior

Innovation

• uniform diffusion language model
• from-scratch pretraining
• open large language model
• non-autoregressive generation
• diffusion-based text generation