Nemotron-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context

2026-06-24

AI Summary

This work addresses the inherent trade-off in existing diffusion language models, where a single shared network is tasked with both contextual modeling and iterative denoising, leading to mutual interference between these objectives. To resolve this, the authors propose TwoTower, a novel architecture that structurally decouples the two functions for the first time: a frozen autoregressive context tower efficiently encodes contextual information, while a trainable bidirectional block-attention diffusion tower focuses exclusively on denoising, with cross-attention mechanisms enabling information fusion between the towers. Built upon a 30B-parameter hybrid Mamba-Transformer mixture-of-experts model (Nemotron-3-Nano-30B-A3B) and trained on 2.1 trillion tokens, the approach achieves 98.7% of the generation quality of a strong autoregressive baseline while delivering a 2.42× improvement in throughput, substantially enhancing generation efficiency without compromising output quality.

Abstract

Diffusion language models offer a promising alternative to autoregressive models due to their potential for parallel and iterative generation. However, existing approaches use a single network for both context representation and iterative denoising, forcing one model to serve both roles and limiting its capacity for either role. We propose TwoTower, a block-wise autoregressive diffusion model that decouples these roles into two towers: a frozen AR context tower that causally processes clean tokens, and a trainable diffusion denoiser tower with bidirectional block attention that refines noisy blocks via cross-attention to the context. Built on Nemotron-3-Nano-30B-A3B, an open-weight 30B hybrid Mamba-Transformer MoE model, and trained on approximately 2.1T tokens, Nemotron-TwoTower retains 98.7% of the autoregressive baseline's quality while offering 2.42X higher wall-clock generation throughput. We release the code and model weights at https://huggingface.co/collections/nvidia/nemotron-twotower.

Problem

• diffusion language models
• autoregressive models
• context representation
• iterative denoising
• model capacity

Innovation

• TwoTower
• diffusion language model
• autoregressive context
• decoupled architecture
• block-wise generation