MineDraft: A Framework for Batch Parallel Speculative Decoding
Standard speculative decoding suffers from limited inference acceleration due to the strict serial dependency between draft generation and verification. This work proposes MineDraft, a novel batch-parallel speculative decoding framework that introduces a dual-batch pipelined scheduling mechanism, overlapping draft generation for one batch of requests with the verification phase of another. Furthermore, MineDraft incorporates a cooperative verification protocol between the draft and target models to ensure output correctness while significantly improving hardware utilization. Integrated into the vLLM system, the proposed approach reduces end-to-end latency by up to 39% and achieves a throughput improvement of up to 75% compared to standard speculative decoding.