Energy-Structured Low-Rank Adaptation for Continual Learning

2026-05-26

AI Summary

This work addresses the challenges of knowledge compression and inter-task interference in continual learning, which are exacerbated by energy dispersion across model parameters. To mitigate these issues, the authors propose Energy-concentrated and Ordered Low-Rank Adaptation (E²-LoRA), a novel mechanism that analyzes the low-rank structure of output feature drift. By preserving critical parameters along principal component directions to minimize reconstruction error, E²-LoRA explicitly compresses and orders accumulated knowledge into leading singular subspaces, thereby freeing model capacity for subsequent tasks. Coupled with a dynamic rank allocation strategy, the method jointly optimizes knowledge retention and model plasticity. Extensive experiments on multiple continual learning benchmarks demonstrate that E²-LoRA significantly outperforms existing approaches, achieving state-of-the-art performance.

Abstract

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose \textbf{E}nergy-Concentrated and \textbf{E}nergy-Ordered \textbf{Lo}w-\textbf{R}ank \textbf{A}daptation (E$^2$-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E$^2$-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E$^2$-LoRA achieves state-of-the-art performance.

Problem

• Continual Learning
• task interference
• energy diffusion
• knowledge compaction
• model capacity

Innovation

• Low-Rank Adaptation
• Continual Learning
• Energy Concentration
• Dynamic Rank Allocation
• Output Feature Drift