How Transparent is DiffusionGemma?

2026-06-18

AI Summary

This study addresses the opacity of reasoning in the DiffusionGemma model within continuous latent spaces by decomposing it into two dimensions: variable transparency and algorithmic transparency. The authors introduce an interpretable token bottleneck to map information flow across denoising steps and combine intermediate state analysis, denoising trajectory tracing, and monitorability evaluation to uncover novel phenomena—namely, non-sequential reasoning, token-sequence ambiguity, and intermediate contextual inference—in diffusion models for the first time. This approach dramatically reduces the model’s opaque serial depth from 28.6× to just 1.1× that of Gemma-4 while preserving comparable monitoring capability, thereby substantially enhancing the interpretability of diffusion-based language models.

Abstract

LLM reasoning transparency is a critical affordance for understanding model decisions, mitigating misuse and misalignment, and debugging surprising model behaviors. However, DiffusionGemma performs a larger fraction of its computation in a continuous latent space; does this make its reasoning less transparent? We study this question by decomposing transparency into two components: variable transparency, whether we understand intermediate snapshots of a model's computational state; and algorithmic transparency, whether we can use these snapshots to reconstruct the process by which the model arrived at its outputs. Naively, DiffusionGemma has poor variable transparency: its opaque serial depth, the amount of serial computation that occurs in between interpretable model states, seems at first 28.6X higher than the corresponding autoregressive Gemma 4 model. However, we show that we can map the information flowing between denoising steps through an interpretable token bottleneck with no decrease in downstream performance. Treating these intermediate states as interpretable reduces the opaque serial depth to just 1.1X that of Gemma 4. Algorithmic transparency is harder for diffusion models than for autoregressive models because all token predictions in the canvas can change at every denoising step, giving the model the power to implement complicated distributed algorithms during the denoising process. To begin bridging this gap, we conduct a suite of interpretability case studies, uncovering initial evidence of novel diffusion-specific phenomena such as non-chronological reasoning, token and sequence smearing, and intermediate-context reasoning. Finally, we test monitorability, a key application of transparency that measures whether model outputs are useful for downstream tasks. We find that DiffusionGemma is similarly monitorable to Gemma 4.

Problem

• transparency
• diffusion models
• reasoning
• interpretability
• latent space

Innovation

• diffusion language models
• reasoning transparency
• interpretable token bottleneck
• opaque serial depth
• algorithmic transparency