Learning New Facts with QLoRA: An Acquisition-Retention Frontier

2026-08-26

AI Summary

研究通过比较QLoRA与全微调在学习新事实时的性能，探讨了适配器容量对预训练能力保留的影响，揭示了参数更新量与知识获取-保持之间的权衡。

Abstract

Parameter-efficient fine-tuning is often assumed to preserve pretrained capabilities because it updates only a small number of parameters. We show that this assumption depends strongly on adapter capacity. We study factual acquisition in a controlled OpenStreetMap-derived benchmark where Qwen3-4B must acquire anonymized geographic associations while retaining unrelated capabilities. Comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) at ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition--retention frontier. Low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at an increasing cost in performance on unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime. Distributional, weight-space, and spectral diagnostics mirror this behavioral trade-off, with higher-rank QLoRA moving farther from the pretrained model. A separate math adaptation experiment shows a weaker frontier, suggesting that the effect is most pronounced when adaptation must install new factual associations rather than reinforce skills already supported by pretraining. Code and data are available at https://github.com/zhngstl/new_facts_forgetting.

Problem

• parameter-efficient fine-tuning
• factual acquisition
• adapter capacity
• acquisition-retention frontier
• pretrained capabilities

Innovation

• Parameter-efficient Fine-tuning
• Quantized Low-rank Adaptation (QLoRA)
• Acquisition-Retention Frontier
• Adapter Capacity