Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion LMs

2026-07-01

AI Summary

This work addresses the tendency of continuous diffusion language models to generate repetitive text, a flaw exacerbated by the commonly used Gen-PPL metric, which erroneously interprets repetition as high quality, thereby distorting evaluation. The study is the first to identify the root cause as a one-dimensional contracting attractor inherent in the self-conditioning mechanism. To mitigate this issue, the authors propose ACE (Attractor Correction at Evaluation), a lightweight inference-time correction method that estimates and suppresses the attractor direction without requiring labeled data. ACE is broadly compatible across multiple models and samplers, effectively reducing repetition rates to near-human levels while preserving generation quality. Moreover, it achieves this with 1.5–5× lower computational overhead compared to existing approaches.

Abstract

Continuous diffusion language models such as ELF report record-low generative perplexity (Gen-PPL). We find a catch: these models repeat far more than human text, and Gen-PPL rewards rather than penalizes that repetition, so its low scores overstate quality. Strip the repetition and ELF-B's Gen-PPL rises from $19.5$ to $27.7$; the smallest model even posts the best Gen-PPL because it repeats most. We trace the repetition to its source: a contractive attractor along a \emph{single direction} in the self-conditioning feedback loop, the loop that feeds each step's clean estimate into the next. Because the failure is one-dimensional, a one-dimensional fix suffices, and we propose one. \textbf{ACE} (Attractor-Contrast-Escape) subtracts that single, label-free direction from the feedback at each step. Estimated once on the $105$M model, the direction cuts repetition to near the human level while keeping quality competitive, and transfers near-unchanged to the $342$M and $652$M models and across samplers; the same recipe recovers useful directions on other architectures. Since Gen-PPL itself rewards repetition, we instead measure the compute each fix needs to produce human-clean text, where ACE is $1.5$--$5\times$ cheaper.

Problem

• repetition
• generative perplexity
• diffusion language models
• self-conditioning
• attractor

Innovation

• continuous diffusion language models
• repetition artifact
• one-dimensional attractor
• self-conditioning feedback
• ACE (Attractor-Contrast-Escape)