Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models

2026-04-29

AI Summary

This work addresses the limitations of existing distillation methods for diffusion-based large language models (dLLMs), which are constrained by same-architecture assumptions and struggle to transfer knowledge across heterogeneous architectures, attention mechanisms, and tokenizers. To overcome this, the authors propose TIDE, the first framework enabling cross-architecture dLLM distillation. TIDE introduces three key components: TIDAL for noise-aware dynamic distillation scheduling, CompDemo to enhance contextual prediction under high masking rates, and Reverse CALM, which aligns heterogeneous tokenizers via bounded-gradient reverse block-wise likelihood matching. Experiments distill 8B dense and 16B MoE teacher models into a 0.6B student model, achieving an average improvement of 1.53 points across eight benchmarks and a HumanEval code generation score of 48.78—substantially outperforming autoregressive baselines (32.3).

Abstract

Diffusion large language models (dLLMs) offer parallel decoding and bidirectional context, but state-of-the-art dLLMs require billions of parameters for competitive performance. While existing distillation methods for dLLMs reduce inference steps within a single architecture, none address cross-architecture knowledge transfer, in which the teacher and student differ in architecture, attention mechanism, and tokenizer. We present TIDE, the first framework for cross-architecture dLLM distillation, comprising three modular components: (1) TIDAL, which jointly modulates distillation strength across training progress and diffusion timestep to account for the teacher's noise-dependent reliability; (2) CompDemo, which enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking; and (3) Reverse CALM, a cross-tokenizer objective that inverts chunk-level likelihood matching, yielding bounded gradients and dual-end noise filtering. Distilling 8B dense and 16B MoE teachers into a 0.6B student via two heterogeneous pipelines outperforms the baseline by an average of 1.53 points across eight benchmarks, yielding notable gains in code generation, where HumanEval scores reach 48.78 compared to 32.3 for the AR baseline.

Problem

• diffusion large language models
• cross-architecture distillation
• knowledge transfer
• model compression
• heterogeneous architectures

Innovation

• cross-architecture distillation
• diffusion language models
• TIDE framework
• noise-aware distillation
• tokenizer-agnostic learning