A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios

2025-10-31

AI Summary

This paper addresses the fundamental trade-off among skill acquisition, knowledge integration, and catastrophic forgetting in large language model (LLM) adaptation under data-scarce conditions. We systematically compare three representative adaptation paradigms: supervised fine-tuning (SFT), low-rank adaptation (LoRA), and in-context learning (ICL). To characterize their behavioral differences, we propose a two-dimensional analytical framework—“Skill Acquisition vs. Knowledge Integration”—and quantitatively evaluate each method’s ability to improve task-specific performance while preserving foundational capabilities. Empirical results demonstrate that LoRA achieves the optimal balance: it effectively acquires new skills with limited data while robustly retaining general-purpose abilities. In contrast, SFT induces significant catastrophic forgetting, whereas ICL excels at injecting factual knowledge but fails to support transfer of complex reasoning skills. Our study provides both theoretical insights and empirical guidance for selecting lightweight LLM adaptation strategies in resource-constrained settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The remarkable capabilities of Large Language Models (LLMs) often need to be tailored for specific applications, requiring the integration of new knowledge or the acquisition of new skills. While full fine-tuning is a powerful adaptation method, it is computationally expensive and can lead to a degradation of general reasoning abilities, a phenomenon known as catastrophic forgetting. A range of alternative techniques exists, each with its own trade-offs. In-Context Learning (ICL) is fast but limited by context length, while Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) offer a middle ground by minimizing parameter changes. However, the challenge of catastrophic forgetting persists, raising questions about the best adaptation strategy for a given task. This paper presents a comparative analysis of Supervised Finetuning (SFT), LoRA, and ICL in data-scarce scenarios. We find that LoRA provides the most effective balance, successfully instilling new skills with minimal impact on the base model's general knowledge. In contrast, while SFT excels at skill acquisition, it is highly susceptible to catastrophic forgetting. ICL is effective for incorporating factual knowledge but struggles with complex skills. Our findings offer a practical framework for selecting an LLM adaptation strategy. We highlight the critical distinction between skill acquisition and knowledge integration, clarify the trade-offs between task-specific performance and the preservation of general capabilities.

Problem

• Comparing LLM adaptation methods in data-limited scenarios
• Evaluating trade-offs between skill acquisition and knowledge preservation
• Addressing catastrophic forgetting during model fine-tuning processes

Innovation

• LoRA balances skill acquisition with minimal forgetting
• SFT acquires skills but causes catastrophic forgetting
• ICL integrates knowledge but struggles with complex skills