Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning

2026-06-29

AI Summary

Full fine-tuning of large language models often degrades their pre-existing capabilities, and current approaches relying on proxy metrics in weight space struggle to precisely preserve functionally relevant directions. This work proposes FORA, a function-space preservation mechanism that shifts capability protection from weight space to function space. FORA constructs a right projection matrix from the principal components of input activation covariances and combines it with the left projection derived from weight SVD, thereby structurally disentangling update paths for retaining original capabilities and learning new tasks. By imposing orthogonality constraints within capability-induced activation subspaces, FORA significantly outperforms weight-space projection and standard regularization strategies. Evaluated on Qwen3-1.7B, it effectively balances retention of translation and mathematical reasoning abilities with new task acquisition, exhibiting only minor trade-offs in mathematical retention scenarios.

Abstract

Full fine-tuning adapts large language models to new tasks but can erode capabilities they already possess. Existing remedies protect through proxies such as parameter distances, importance penalties, output matching, or dominant singular directions of the weights, but none directly asks which activation directions the preserved capability relies on. We argue that a capability is characterized more faithfully by the activation subspace it induces than by the singular geometry of the weight matrix, and develop function-space protection, instantiated as FORA (Function-space Orthogonal Residual Adaptation). From label-free calibration inputs, FORA estimates, per layer, the principal directions $Q$ of the input-activation covariance and forms a right projector $P_Q = I - QQ^T$. Paired with a left projector $P_U$ from the weight SVD, the update is $ΔW = P_U M P_Q + U_2 D_δ V_2^T$: a high-capacity branch structurally barred from reading capability-relevant function directions, plus a narrow spectral channel for controlled plasticity. The construction extends to parameter-efficient adaptation via $M \to (α/r) BA$. Across three settings on Qwen3-1.7B, including COGS and GSM8K learned while preserving translation and translation learned while preserving math, FORA consistently improves preservation over weight-space projection and standard regularization, with only a small new-task trade-off in the math-preservation setting. A controlled ablation isolating the projection source shows that the advantage comes not from projection itself, but from projecting onto capability-derived rather than weight-derived directions. Code is available at https://github.com/zrui239/FORA.

Problem

• capability preservation
• fine-tuning
• function-space
• activation subspace
• catastrophic forgetting

Innovation

• function-space protection
• capability-preserving fine-tuning
• activation subspace
• orthogonal residual adaptation
• parameter-efficient adaptation