S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain

2026-05-08

AI Summary

Existing Fourier transform–based parameter-efficient fine-tuning methods assume that weight updates are spectrally sparse; however, empirical observations reveal that their spectral energy is in fact uniformly distributed, limiting performance. This work proposes S2FT, which first uncovers the inherently non-sparse nature of weight-update spectra and introduces an invertible transformation based on row–column permutation to map weight changes into a latent space exhibiting local smoothness. This structural prior induces sparsity in the transformed spectral domain. By integrating this spectral prior with a nearest-neighbor search strategy, S2FT achieves superior fine-tuning performance while updating only 0.08% of the model parameters—outperforming current state-of-the-art approaches.

Abstract

Parameter Efficient Fine-Tuning (PEFT) is a key technique for adapting a large pretrained model to downstream tasks by fine-tuning only a small number of parameters. Recent methods based on Fourier transforms have further reduced the fine-tuned parameters scale by only fine-tuning a few spectral coefficients. Its basic assumption is that the weight change δW is a spatial-domain matrix with a sparse spectrum. However, in this paper, we observe that the spectrum of weight change is not sparse, but instead distributed like power-uniform. This fact implies that fine-tuning only a few spectral coefficients is insufficient to accurately model the weight change with uniform spectrum. To address this issue, we propose to seek an invertible transformation that can transform a latent spatial-domain matrix with sparse spectrum to the weight change, and then perform PEFT on such sparse spectrum domain with few spectral coefficients, called S2FT. To seek such transformation, we first pre-estimate a coarse weight change as a prior. Then, inspired by that sparse spectrum often correspond to locally smooth spatial structures, we regard this transformation as a row and column rearrangement operation on the pre-estimated weight change that smooth spatial structures while keep the structure information of neurons. Finally, we propose to solve the rearrangement search problem in a simple nearest neighbor search manner, thereby obtaining the invertible transformation. Extensive results show our S2FT achieves superior performance by only using 0.08% training parameters.

Problem

• Parameter-Efficient Fine-Tuning
• Sparse Spectrum
• Weight Change
• Fourier Transform
• Spectral Coefficients

Innovation

• Parameter-Efficient Fine-Tuning
• Sparse Spectrum
• Invertible Transformation
• Fourier Domain
• Weight Rearrangement