Fine-Tuned In-Context Learners for Efficient Adaptation

2025-12-22

AI Summary

To address the performance saturation of in-context learning (ICL) with increasing data and the poor generalization of fine-tuning under low-resource settings, this paper proposes an ICL-aware fine-tuning paradigm. It explicitly incorporates k-shot ICL structure into end-to-end fine-tuning by dynamically injecting task-relevant in-context examples into training instances. This work establishes, for the first time, a unified modeling framework bridging fine-tuning and ICL. We further design a validation-free prequential evaluation mechanism to enable efficient hyperparameter selection in low-resource scenarios. Experiments across multiple low-resource tasks demonstrate that our method consistently outperforms both standard fine-tuning and pure ICL baselines: it yields substantial gains in few-shot regimes and exhibits more stable performance improvements as training data scale increases.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

When adapting large language models (LLMs) to a specific downstream task, two primary approaches are commonly employed: (1) prompt engineering, often with in-context few-shot learning, leveraging the model's inherent generalization abilities, and (2) fine-tuning on task-specific data, directly optimizing the model's parameters. While prompt-based methods excel in few-shot scenarios, their effectiveness often plateaus as more data becomes available. Conversely, fine-tuning scales well with data but may underperform when training examples are scarce. We investigate a unified approach that bridges these two paradigms by incorporating in-context learning directly into the fine-tuning process. Specifically, we fine-tune the model on task-specific data augmented with in-context examples, mimicking the structure of k-shot prompts. This approach, while requiring per-task fine-tuning, combines the sample efficiency of in-context learning with the performance gains of fine-tuning, leading to a method that consistently matches and often significantly exceeds both these baselines. To perform hyperparameter selection in the low-data regime, we propose to use prequential evaluation, which eliminates the need for expensive cross-validation and leverages all available data for training while simultaneously providing a robust validation signal. We conduct an extensive empirical study to determine which adaptation paradigm - fine-tuning, in-context learning, or our proposed unified approach offers the best predictive performance on a concrete data downstream-tasks.

Problem

• Bridging prompt engineering and fine-tuning for LLM adaptation
• Enhancing sample efficiency and performance in low-data scenarios
• Eliminating expensive cross-validation via prequential evaluation

Innovation

• Fine-tuning with in-context examples for adaptation
• Prequential evaluation for hyperparameter selection
• Unified approach combining in-context learning and fine-tuning