Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

2026-07-10

AI Summary

Full-parameter fine-tuning of large language models incurs substantial computational costs and demands extensive memory and storage. To address this, this work proposes Supra, a hybrid adapter that integrates sparse fine-tuning with low-rank adaptation (LoRA). Supra is the first to leverage activation-weighted magnitude scores—such as those from Wanda—to guide the selection of a fixed, sparse trainable parameter subset, and introduces a budget allocation strategy to jointly optimize sparse updates and low-rank structures. Evaluated on the Math17K benchmark using Llama-3.2-1B and Meta-Llama-3-8B, Supra significantly outperforms existing parameter-efficient fine-tuning methods, demonstrating the effectiveness of pruning-based heuristics in enhancing fine-tuning efficiency and performance.

Abstract

Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.

Problem

• parameter-efficient fine-tuning
• sparse adaptation
• large language models
• activation-aware pruning
• computational cost

Innovation

• sparse fine-tuning
• activation-aware pruning
• parameter-efficient adaptation
• LoRA
• Wanda