Take Off the Training Wheels! Progressive In-Context Learning for Effective Alignment

2025-03-13

Conference on Empirical Methods in Natural Language Processing

AI Summary

Existing in-context learning (ICL) research primarily targets classification and simple generation tasks, offering limited support for complex alignment tasks. Method: This paper first identifies that delimiter tokens in ICL implicitly encode task-function representations; leveraging this insight, we propose PICA—a training-free, two-stage progressive contextual alignment framework. Stage one extracts delimiter-driven ICL vectors via few-shot prompting; stage two reuses these vectors for zero-shot inference-time alignment. Grounded in Transformer attention analysis, PICA employs decoupled context construction and vector injection—requiring no parameter updates. Contribution/Results: On multi-task alignment benchmarks, PICA matches fine-tuning performance while substantially outperforming standard ICL. It achieves a 5.45× inference speedup and improves alignment accuracy by +6.57 points, significantly reducing computational overhead.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent studies have explored the working mechanisms of In-Context Learning (ICL). However, they mainly focus on classification and simple generation tasks, limiting their broader application to more complex generation tasks in practice. To address this gap, we investigate the impact of demonstrations on token representations within the practical alignment tasks. We find that the transformer embeds the task function learned from demonstrations into the separator token representation, which plays an important role in the generation of prior response tokens. Once the prior response tokens are determined, the demonstrations become redundant. Motivated by this finding, we propose an efficient Progressive In-Context Alignment (PICA) method consisting of two stages. In the first few-shot stage, the model generates several prior response tokens via standard ICL while concurrently extracting the ICL vector that stores the task function from the separator token representation. In the following zero-shot stage, this ICL vector guides the model to generate responses without further demonstrations. Extensive experiments demonstrate that our PICA not only surpasses vanilla ICL but also achieves comparable performance to other alignment tuning methods. The proposed training-free method reduces the time cost (e.g., 5.45×) with improved alignment performance (e.g., 6.57+). Consequently, our work highlights the application of ICL for alignment and calls for a deeper understanding of ICL for complex generations. The code will be available at https://github.com/HITsz-TMG/PICA.

Problem

• Explores In-Context Learning for complex generation tasks.
• Proposes Progressive In-Context Alignment (PICA) method.
• Reduces time cost and improves alignment performance.

Innovation

• Progressive In-Context Alignment (PICA) method
• Extracts ICL vector from separator token
• Guides response generation without demonstrations