Compress Then Adapt? No, Do It Together via Task-aware Union of Subspaces

2026-05-04

AI Summary

This work addresses the suboptimal alignment between compression subspaces and downstream task objectives in conventional parameter-efficient fine-tuning (PEFT) pipelines, where low-rank compression and adaptation are performed sequentially, often leading to inefficient parameter allocation. To overcome this limitation, the authors propose JACTUS, a novel framework that jointly optimizes compression and task adaptation within a unified, task-aware subspace. JACTUS leverages a calibration set to estimate the covariance of inputs and pre-activation gradients, constructs an orthogonal subspace ensemble, and performs projection-based low-rank approximation with margin-gain-guided rank allocation, training only a compact core matrix. Experiments demonstrate that JACTUS achieves average accuracies of 89.2% on ViT-Base (with 80% parameter retention) and 80.9% on Llama2-7B, significantly outperforming both full-parameter PEFT and sequential compress-then-finetune baselines.

Abstract

Adapting large pretrained models to diverse tasks is now routine, yet the two dominant strategies of parameter-efficient fine-tuning (PEFT) and low-rank compression are typically composed in sequence. This decoupled practice first compresses and then fine-tunes adapters, potentially misaligning the compressed subspace with downstream objectives and squandering a global parameter budget. To overcome this limitation, we introduce JACTUS (Joint Adaptation and Compression with a Task-aware Union of Subspaces), a single framework that unifies compression and adaptation. From a small calibration set, JACTUS estimates input and pre-activation gradient covariances, forms their orthogonal union with the pretrained weight subspace, performs a projected low-rank approximation inside this union, allocates rank globally by marginal gain per parameter, and trains only a compact core matrix. This explicitly mitigates the potential misalignment between the compressed subspace and downstream objectives by coupling the directions preserved for compression with those required for adaptation, yielding a deployable low-rank model that avoids retaining full frozen weights while enabling fast and robust tuning. On vision, JACTUS attains an average 89.2% accuracy on ViT-Base across eight datasets at 80% retained parameters, surpassing strong 100% PEFT baselines (e.g., DoRA 87.9%). On language, JACTUS achieves an 80.9% average on Llama2-7B commonsense QA at the same 80% retained-parameter budget, outperforming 100% PEFT (e.g., DoRA 79.7%) and exceeding prior compress-then-finetune pipelines under the same ratained-parameter budget. We will release code.

Problem

• parameter-efficient fine-tuning
• low-rank compression
• subspace misalignment
• pretrained model adaptation
• parameter budget

Innovation

• joint adaptation and compression
• task-aware subspace
• low-rank approximation
• parameter-efficient fine-tuning
• subspace union