CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA

2026-07-11

AI Summary

This work addresses the activation memory bottleneck that arises when fine-tuning large pretrained models under limited memory constraints using Low-Rank Adaptation (LoRA). The authors propose CARE-LoRA, which introduces, for the first time, a data-aware low-rank activation compression and reconstruction mechanism. During forward propagation, compressed activations replace full inputs, while during backward propagation, a lightweight computation reconstructs a matrix to recover gradient signals, thereby preserving the full trainability of LoRA parameters. This approach substantially reduces activation memory overhead while achieving performance on par with or even superior to standard LoRA across diverse models and downstream tasks, establishing a new paradigm for parameter-efficient fine-tuning.

Abstract

As the scale of large pre-trained models continues to grow, fine-tuning them under limited memory budgets has become increasingly challenging. Low-Rank Adaptation (LoRA), currently one of the most widely adopted parameter-efficient fine-tuning (PEFT) methods, mitigates this challenge by optimizing only low-rank adaptation matrices, thereby greatly reducing the number of trainable parameters. With the parameter overhead substantially reduced, the activations retained for backpropagation have emerged as the primary remaining memory bottleneck during LoRA fine-tuning. To address this, we propose CARE-LoRA, a data-aware Compressed Activation REconstruction framework. By exploiting the inherent projection structure of LoRA, CARE-LoRA replaces the full input activation with the low-rank compressed activation naturally produced by the LoRA branch. It further computes a lightweight reconstruction matrix during the forward pass with negligible additional computation cost, which is used during backpropagation to reconstruct the gradient signal, thereby keeping LoRA matrices fully trainable. Extensive experiments across diverse models and downstream tasks demonstrate that, while substantially reducing the overall memory footprint, CARE-LoRA achieves competitive or even superior performance compared with standard LoRA and representative LoRA variants. Our code is publicly available at https://github.com/fishandyu/CARE-LoRA .

Problem

• memory-efficient fine-tuning
• activation memory bottleneck
• Low-Rank Adaptation
• parameter-efficient fine-tuning
• large language models

Innovation

• LoRA
• activation compression
• memory-efficient fine-tuning
• parameter-efficient fine-tuning
• gradient reconstruction