Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B

2026-07-05

AI Summary

This work addresses the challenge of achieving low-latency inference for the Qwen3.5-4B model on resource-constrained NVIDIA A10G GPUs by proposing an efficient approach that integrates quantization with speculative decoding. The method employs quantization-aware distillation to recover model accuracy and introduces a two-stage training strategy to construct a block-wise diffusion draft model specifically tailored for the quantized target model. Furthermore, the draft model is enhanced with quantization and sliding window attention to improve long-context decoding efficiency while maintaining a high token acceptance rate. Experimental results demonstrate that, under strict quality constraints, the proposed method achieves an average speedup of 6.98× over the baseline and secured third place in a related competition.

Abstract

This report describes our approach to the Efficient Qwen Competition, where the goal is to enable low-latency serving of Qwen3.5-4B on a resource-constrained NVIDIA A10G GPU. Our system combines a quantized target model with speculative decoding. To recover accuracy, we apply quantization-aware distillation to the target model while retaining the original quantization grid. To speed up decoding, a block-diffusion drafter specialized for the quantized target model is trained using a two-stage procedure: first learning from the high-precision target and then adapting to the low-precision target. Because the drafter is invoked at every speculative decoding step, we further reduce its overhead with quantization and sliding-window attention, preserving draft-token acceptance while improving long-context decoding latency. As a result, our submission achieves a 6.978$\times$ average speedup over the baseline while satisfying the required quality thresholds, ranking 3rd overall. We hope these results provide useful insights for practical LLM inference. The code and resources are available at https://github.com/nota-github/adaptfm-quant-dflash

Problem

• efficient inference
• large language model
• quantization
• speculative decoding
• low-latency serving

Innovation

• quantization-aware distillation
• speculative decoding
• block-diffusion drafter
• sliding-window attention
• low-precision LLM inference