Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

2026-04-15

AI Summary

Existing instruction-tuning datasets often conflate world knowledge acquired during pretraining with the instruction-following capabilities developed during post-training, thereby limiting fine-tuning effectiveness. This work proposes CoDIT (Contrastive Decoding for Instruction Tuning), a method that leverages contrastive decoding between a post-trained model and its pretrained counterpart to suppress shared world knowledge and amplify pure instruction-following behavior. CoDIT achieves, for the first time, the distillation of a “chat vector” from parameter space into textual space, effectively disentangling and transferring instruction-following ability in a manner compatible across diverse model architectures. Models trained on datasets constructed via CoDIT substantially outperform those trained on directly generated data or existing public instruction-tuning benchmarks, demonstrating significantly enhanced instruction-following performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Diffusion Models for Vision

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.

Problem

• instruction tuning
• large language models
• pre-trained knowledge
• instruction-following capabilities
• response generation

Innovation

• contrastive decoding
• instruction tuning
• knowledge disentanglement
• response generation
• capability distillation