Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation

2026-03-10

AI Summary

This work addresses the performance degradation of speculative decoding caused by the mismatch between a target large language model and its draft model after domain-specific fine-tuning, a problem exacerbated by the prohibitive cost of retraining a dedicated draft model for each fine-tuned variant. To overcome this, the authors propose EDA, an efficient draft model adaptation framework that decouples shared and private components, employs a target-model-guided data regeneration strategy, and incorporates a high-value sample selection mechanism. This approach enables effective adaptation at both the parameter and data levels without requiring full retraining. Experimental results demonstrate that EDA substantially restores speculative decoding performance across multiple fine-tuned models, achieving higher average accepted lengths than baseline methods while significantly reducing training overhead.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Speculative decoding accelerates LLM inference but suffers from performance degradation when target models are fine-tuned for specific domains. A naive solution is to retrain draft models for every target model, which is costly and inefficient. To address this, we introduce a parameter- and data-efficient framework named Efficient Draft Adaptation, abbreviated as EDA, for efficiently adapting draft models. EDA introduces three innovations: (1) a decoupled architecture that utilizes shared and private components to model the shared and target-specific output distributions separately, enabling parameter-efficient adaptation by updating only the lightweight private component;(2) a data regeneration strategy that utilizes the fine-tuned target model to regenerate training data, thereby improving the alignment between training and speculative decoding, leading to higher average acceptance length;(3) a sample selection mechanism that prioritizes high-value data for efficient adaptation. Our experiments show that EDA effectively restores speculative performance on fine-tuned models, achieving superior average acceptance lengths with significantly reduced training costs compared to full retraining. Code is available at https://github.com/Lyn-Lucy/Efficient-Draft-Adaptation.

Problem

• speculative decoding
• draft model alignment
• fine-tuned LLMs
• parameter efficiency
• data efficiency

Innovation

• speculative decoding
• parameter-efficient adaptation
• data regeneration
• decoupled architecture
• sample selection