Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

2026-01-21

AI Summary

This study investigates whether autoregressive models, after post-training conversion into masked diffusion models, genuinely acquire bidirectional reasoning capabilities or merely repurpose their original heuristic strategies. Employing circuit analysis, the authors systematically compare the internal computational pathways of both model types across local causal and global planning tasks. They uncover, for the first time, a task-dependent mechanistic shift induced by post-training: while local tasks retain the original pathways, global tasks trigger a reconfiguration of early layers, effecting a semantic transition from localized specialization to distributed integration. These findings demonstrate that diffusion-based post-training not only fine-tunes parameters but fundamentally reorganizes the internal computational architecture, endowing the model with non-sequential global planning abilities that transcend the inherent limitations of autoregressive generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Post-training pretrained Autoregressive models (ARMs) into Masked Diffusion models (MDMs) has emerged as a cost-effective strategy to overcome the limitations of sequential generation. However, the internal algorithmic transformations induced by this paradigm shift remain unexplored, leaving it unclear whether post-trained MDMs acquire genuine bidirectional reasoning capabilities or merely repackage autoregressive heuristics. In this work, we address this question by conducting a comparative circuit analysis of ARMs and their MDM counterparts. Our analysis reveals a systematic"mechanism shift"dependent on the structural nature of the task. Structurally, we observe a distinct divergence: while MDMs largely retain autoregressive circuitry for tasks dominated by local causal dependencies, they abandon initialized pathways for global planning tasks, exhibiting distinct rewiring characterized by increased early-layer processing. Semantically, we identify a transition from sharp, localized specialization in ARMs to distributed integration in MDMs. Through these findings, we conclude that diffusion post-training does not merely adapt model parameters but fundamentally reorganizes internal computation to support non-sequential global planning.

Problem

• mechanism shift
• autoregressive models
• masked diffusion models
• bidirectional reasoning
• post-training

Innovation

• mechanism shift
• masked diffusion language models
• autoregressive models
• circuit analysis
• global planning