EpiCoDe: Boosting Model Performance Beyond Training with Extrapolation and Contrastive Decoding

2025-06-04

AI Summary

To address the performance degradation of fine-tuning large language models (LLMs) on downstream tasks under data-scarce conditions, this paper proposes a training-free framework that synergistically integrates extrapolation enhancement and contrastive decoding. Methodologically, we are the first to jointly model extrapolation and contrastive decoding, introducing a logit-score-difference quantification mechanism and providing the first theoretical analysis of contrastive decoding’s effectiveness under low-data regimes. Experiments span three representative downstream tasks and four mainstream LLMs, demonstrating that our approach consistently and significantly outperforms existing methods—achieving average improvements of 3.2–7.8 percentage points. Our core contributions are threefold: (1) establishing the first extrapolation–contrastive decoding co-design paradigm; (2) grounding contrastive decoding in formal theoretical interpretability; and (3) empirically validating its strong generalization capability in data-scarce settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

The remarkable performance of Large language models (LLMs) relies heavily on the availability of abundant high-quality training data. However, the high cost of acquiring annotated data often prevents models from obtaining capabilities to tackle downstream tasks. In this paper, we introduce a novel method, EpiCoDe that boosts model performance in data-scarcity scenarios without extra training. We first employ model extrapolation to enhance a finetuned model with its inferior version, and then adopt contrastive decoding to further reduce predicted errors, by comparing the logit scores given by the extrapolated and the vanilla finetuned model. Experiments across three tasks over four different LLMs show that EpiCoDe consistently outperforms existing methods with significant and robust improvement. We also propose a new theoretical framework to reveal the mechanism behind contrastive decoding in data-scarcity scenarios, which further helps us better understand the effectiveness of EpiCoDe.

Problem

• Enhancing model performance without extra training data
• Reducing predicted errors using contrastive decoding
• Improving LLMs in data-scarcity scenarios via extrapolation

Innovation

• Extrapolation enhances model with inferior version
• Contrastive decoding reduces predicted errors
• Theoretical framework explains contrastive decoding mechanism