DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning

2026-04-13

AI Summary

Existing task vector methods rely on fine-tuning or intrusive internal modifications, limiting their flexibility and scalability. This work proposes DeCoVec—a training-free, non-intrusive framework that, for the first time, constructs task vectors in the decoding space through in-context learning. DeCoVec leverages the logit distribution discrepancy between few-shot and zero-shot prompt outputs to guide generation, requiring no weight updates, auxiliary models, or additional input tokens. Experiments across seven large language models ranging from 0.5B to 9B parameters demonstrate that DeCoVec significantly outperforms few-shot baselines, achieving up to a 5.50% absolute improvement in average accuracy. It effectively mitigates generation degeneration and logical errors while exhibiting strong robustness to the ordering of in-context examples.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Task vectors, representing directions in model or activation spaces that encode task-specific behaviors, have emerged as a promising tool for steering large language models (LLMs). However, existing approaches typically require fine-tuning or invasive manipulation of internal states, limiting their flexibility and scalability. We propose \textsc{DeCoVec} (Decoding Space based Task Vector), a training-free and non-invasive framework that constructs task vectors directly in the \textit{decoding space} by leveraging in-context learning (ICL). Specifically, \textsc{DeCoVec} captures the task essence as the difference between the output logit distributions of few-shot and zero-shot prompts, then steers generation by injecting this vector into the decoding process. Experiments across seven LLMs (0.5B--9B) on TruthfulQA, Math-500, and AQUA-RAT show that \textsc{DeCoVec} consistently outperforms standard few-shot baselines, with gains up to +5.50 average accuracy. Further analysis demonstrates that \textsc{DeCoVec} effectively suppresses generation degeneration and logical flaws while exhibiting strong robustness to demonstration ordering, all without incurring additional input token costs. Our method offers a training-free and non-invasive solution for LLM steering without requiring weight updates or auxiliary models.

Problem

• task vectors
• large language models
• in-context learning
• non-invasive steering
• decoding space

Innovation

• task vector
• decoding space
• in-context learning
• training-free
• non-invasive