Diffutron: A Masked Diffusion Language Model for Turkish Language

2026-03-20

AI Summary

This work addresses the limited performance of non-autoregressive language models on morphologically complex Turkish by proposing Diffutron, a mask-based diffusion language model tailored for Turkish. The approach innovatively integrates mask diffusion modeling with continual pretraining of a multilingual encoder enhanced via LoRA (Low-Rank Adaptation), followed by progressive instruction tuning that transitions from general-purpose to task-specific objectives. Despite its compact scale, Diffutron achieves performance on multiple benchmarks comparable to that of billion-parameter models, demonstrating its efficiency and effectiveness in handling the linguistic complexities of Turkish.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Masked Diffusion Language Models (MDLMs) have emerged as a compelling non-autoregressive alternative to standard large language models; however, their application to morphologically rich languages remains limited. In this paper, we introduce $\textit{Diffutron}$, a masked diffusion language model specifically designed for Turkish. Our approach leverages a resource-efficient training pipeline, starting with LoRA-based continual pre-training of a multilingual encoder on a large-scale corpus. To enable generative capabilities, we employ a progressive instruction-tuning strategy, sequentially adapting the model on general and task-specific instruction sets. Experimental results across comprehensive benchmarks demonstrate that, despite its compact size, our model achieves competitive performance compared to existing multi-billion-parameter baselines. These findings validate the effectiveness of masked diffusion modeling combined with multi-stage tuning for non-autoregressive text generation in Turkish.

Problem

• Masked Diffusion Language Models
• Turkish Language
• Morphologically Rich Languages
• Non-autoregressive Generation

Innovation

• Masked Diffusion Language Model
• LoRA-based continual pre-training
• progressive instruction-tuning
• non-autoregressive generation
• morphologically rich language