PALoRA: Projection-Adaptive LoRA for Preserving Reasoning in Large Language Models

2026-05-23

AI Summary

This work addresses the challenge of balancing plasticity and stability in large language models during knowledge updating. The authors discover that reasoning capabilities rely on the full spectrum of singular directions in multilayer perceptron weight matrices, rather than solely on principal components. Building on this insight, they propose a two-stage parameter-efficient fine-tuning framework: first, Singular Value Fine-tuning (SVF) identifies the critical subspace for reasoning; then, new knowledge is injected via LoRA under orthogonality constraints to minimize interference with existing reasoning abilities. Evaluated on Llama 3.1 8B and Mistral 7B, the method preserves 95% of original reasoning performance on average while maintaining strong factual recall, significantly outperforming existing spectral-domain PEFT approaches with a parameter overhead of less than 0.006%.

Abstract

Efficiently updating Large Language Models (LLMs) with new or evolving factual knowledge remains a central challenge, as even parameter-efficient adaptation can erode previously acquired reasoning abilities. This tension reflects a plasticity-stability dilemma: models must incorporate new knowledge while preserving skill-critical representations. In this work, we study this trade-off through the spectral structure of multilayer perceptron weight matrices. We show, both theoretically and empirically, that information essential for reasoning is not localized only in dominant singular directions, but is instead distributed across the singular spectrum. Motivated by this observation, we introduce PALoRA, a two-stage framework for knowledge injection with reduced interference. PALoRA first trains a Singular Value Fine-Tuning (SVF) expert on a reasoning dataset and uses its learned singular scaling vector as a frozen geometric probe to identify components that are critical for the target skill. It then performs factual knowledge injection with Low-Rank Adaptation (LoRA) under a structural orthogonality constraint, ensuring that updates avoid the identified skill-relevant subspace. Across Llama 3.1 8B and Mistral 7B, and across mathematical, coding, and scientific reasoning benchmarks, PALoRA preserves on average 95% of the SVF expert's reasoning performance while maintaining competitive factual recall. It consistently improves skill retention over prior spectral Parameter-Efficient Fine-Tuning (PEFT) methods while adding less than 0.006% parameter overhead.

Problem

• plasticity-stability dilemma
• reasoning preservation
• knowledge injection
• parameter-efficient fine-tuning
• large language models

Innovation

• PALoRA
• Singular Value Fine-Tuning
• LoRA
• spectral structure
• parameter-efficient fine-tuning