Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation

2026-01-16

AI Summary

This work addresses the challenge that large language models, due to frozen parameters, struggle to effectively incorporate new knowledge and enhance reasoning and decision-making capabilities through supervised fine-tuning (SFT) alone. The authors propose the Parametric Skill Transfer (PaST) framework, which leverages the observation that parameter updates induced by SFT and reinforcement learning (RL) are approximately orthogonal. Building on this insight, PaST introduces a modular mechanism that extracts domain-agnostic skill vectors from a source domain, applies lightweight fine-tuning, and linearly injects them into the target model to enable efficient cross-domain skill transfer. This approach establishes a scalable continual adaptation framework, achieving significant performance gains: +9.9 points over the state-of-the-art self-editing SFT baseline on SQuAD, +8.0 points on LooGLE long-context question answering, and an average +10.3-point improvement in zero-shot tool-use success rate on ToolBench, demonstrating strong generalization capabilities.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) face the"knowledge cutoff"challenge, where their frozen parametric memory prevents direct internalization of new information. While Supervised Fine-Tuning (SFT) is commonly used to update model knowledge, it often updates factual content without reliably improving the model's ability to use the newly incorporated information for question answering or decision-making. Reinforcement Learning (RL) is essential for acquiring reasoning skills; however, its high computational cost makes it impractical for efficient online adaptation. We empirically observe that the parameter updates induced by SFT and RL are nearly orthogonal. Based on this observation, we propose Parametric Skill Transfer (PaST), a framework that supports modular skill transfer for efficient and effective knowledge adaptation. By extracting a domain-agnostic Skill Vector from a source domain, we can linearly inject knowledge manipulation skills into a target model after it has undergone lightweight SFT on new data. Experiments on knowledge-incorporation QA (SQuAD, LooGLE) and agentic tool-use benchmarks (ToolBench) demonstrate the effectiveness of our method. On SQuAD, PaST outperforms the state-of-the-art self-editing SFT baseline by up to 9.9 points. PaST further scales to long-context QA on LooGLE with an 8.0-point absolute accuracy gain, and improves zero-shot ToolBench success rates by +10.3 points on average with consistent gains across tool categories, indicating strong scalability and cross-domain transferability of the Skill Vector.

Problem

• knowledge cutoff
• continual adaptation
• reasoning skills
• supervised fine-tuning
• reinforcement learning

Innovation

• Parametric Skill Transfer
• Reinforcement Learning
• Supervised Fine-Tuning
• Skill Vector
• Continual Adaptation