Fine-Tuning of Transformer models with Frames

2026-08-26

AI Summary

本文提出FrameFT方法，通过在融合框架基础上使用稀疏系数矩阵来更新参数，减少内存占用并提高计算效率，解决了大规模预训练模型微调时的高内存需求问题。

Abstract

Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $ΔW$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.

Problem

• Parameter-Efficient Fine-Tuning
• Low-Rank Adaptation
• memory requirements

Innovation

• Parameter-Efficient Fine-Tuning
• Fusion Frames
• Sparse Coefficient Matrix
• Low Memory Footprint