CSD: Content-aware Speculative Decoding for Efficient Image Generation

2026-06-26

AI Summary

This work addresses the limitations of speculative decoding in autoregressive image generation, where low acceptance rates hinder efficiency and overly permissive acceptance criteria degrade output quality. To reconcile this trade-off, the authors propose Content-aware Speculative Decoding (CSD), the first method to integrate a content-aware mechanism into this paradigm. CSD dynamically adjusts token acceptance thresholds across image regions based on local entropy, relaxing criteria in low-detail areas to boost acceptance rates and accelerate generation. It further incorporates an optimal resampling strategy and a distribution alignment filter to preserve fidelity by ensuring the output distribution remains consistent with that of the target model. Experiments on Lumina-mGPT and Janus-Pro demonstrate that CSD substantially improves both inference speed and generation quality, achieving a synergistic optimization of efficiency and visual fidelity.

Abstract

Speculative decoding (SD) has emerged as a key solution to accelerate the inference of autoregressive models. However, in the field of image generation, it faces the challenge of low acceptance rates, and directly relaxing its criteria leads to degradation in image quality. In this paper, we propose a novel content-aware speculative decoding algorithm, termed CSD, which integrates an entropy-based probability relaxation mechanism with an optimal resampling strategy to enhance the inference efficiency for autoregressive image generation. By leveraging the informational uncertainty inherent in different regions of an image, CSD dynamically adjusts the acceptance probability of candidate tokens, increasing the acceptance rate in low-detail areas to accelerate generation. Moreover, a distribution alignment filter is introduced to ensure the output distribution to be aligned with the target model, which significantly improves the generative quality. Experiments conducted on Lumina-mGPT and Janus-Pro demonstrate that the superiority of the proposed CSD. Our source code is available at https://github.com/aderfebr/CSD.

Problem

• speculative decoding
• image generation
• acceptance rate
• autoregressive models
• generation quality

Innovation

• speculative decoding
• content-aware
• entropy-based relaxation
• distribution alignment
• autoregressive image generation