Prompt Compression in Diffusion Large Language Models: Evaluating LLMLingua-2 on LLaDA

2026-05-18

AI Summary

This study investigates the applicability of prompt compression methods designed for autoregressive models—specifically LLMLingua-2—to diffusion-based language models, using LLaDA-8B as a testbed. Through tasks including mathematical reasoning, summarization, and prompt reconstruction, the authors systematically evaluate the impact of compression on semantic retention and task performance using multidimensional metrics such as exact match, BLEU, ROUGE, and BERTScore. The work reveals, for the first time, an inconsistency between semantic similarity and reasoning stability in diffusion models under compression, identifying information omission as the primary cause of performance degradation. While summarization proves robust to compression, mathematical reasoning suffers significant decline. Notably, BERTScore consistently shows lower recall than precision, indicating loss of critical content rather than semantic drift. These findings advance the development of prompt compression strategies tailored to diffusion mechanisms.

Abstract

Prompt compression reduces inference cost and context length in large language models, but prior evaluations focus primarily on autoregressive architectures. This study investigates whether prompt compression transfers effectively to diffusion large language models (DLLMs) using LLMLingua-2, specifically the 8B-parameter DLLM LLaDA. We evaluate compression performance on GSM8K, DUC2004, and ShareGPT using 250 prompts per dataset at an approximate 2$\times$ compression ratio, across mathematical reasoning, prompt reconstruction, and summarization tasks. Outputs generated from original prompts, compressed prompts, reconstructed prompts, and reconstructed-prompt reasoning were compared using exact-match accuracy, BLEU, ROUGE, and BERTScore. Results show that semantic preservation does not necessarily imply stable downstream behavior in diffusion models. Summarization tasks remained comparatively robust under compression, while mathematical reasoning degraded substantially despite high semantic similarity scores. Reconstruction experiments further showed that semantically similar prompts may still omit reasoning-critical information required for stable denoising. Across tasks, BERTScore recall was consistently lower than precision, suggesting that compression failures are primarily driven by information omission rather than semantic drift. These findings indicate that prompt compression methods designed for autoregressive models do not transfer uniformly to diffusion large language models and motivate the development of diffusion-aware compression strategies.

Problem

• prompt compression
• diffusion large language models
• semantic preservation
• downstream task performance
• information omission

Innovation

• prompt compression
• diffusion large language models
• semantic preservation
• information omission
• LLaDA