SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning

2026-05-20

AI Summary

This work addresses the limitation of Low-Rank Adaptation (LoRA) under tight parameter budgets, where excessively low rank impedes adequate coverage of critical spectral directions in pretrained weight matrices. To overcome this, the authors propose Spectral Mixture of Adaptations (SMoA), which partitions the weight matrix into aligned spectral blocks and introduces Hadamard-product-modulated low-rank branches within each block. This design uniquely integrates spectral block partitioning with Hadamard modulation, substantially expanding the learnable, spectrum-aware update space without increasing the parameter budget, thereby breaking LoRA’s rank–performance trade-off. Experiments demonstrate that SMoA consistently outperforms LoRA and its variants across multiple downstream tasks, achieving significant average performance gains with equal or fewer parameters.

Abstract

As the number of model parameters increases, parameter-efficient fine-tuning (PEFT) has become the go-to choice for tailoring pre-trained large language models. Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning, which is widely used to reduce resource requirements. However, decreasing the rank encounters challenges with limited representational capacity. Theory suggests that LoRA fine-tuning with rank r converges toward the top r singular values of the pre-trained weight matrix. As the rank increases, more principal singular directions are preserved, which generally improves the model's performance. However, a larger rank also introduces more trainable parameters, leading to higher computational cost. To overcome this dilemma, we propose SMoA, a \textbf{S}pectrum \textbf{Mo}dulation \textbf{A}dapter that enlarges the accessible family of spectrum-aware updates under a smaller parameter budget. SMoA partitions the layer into multiple aligned spectral blocks and applies one in-block Hadamard-modulated low-rank branch to each diagonal block, yielding broader coverage of pretrained spectral directions. We provide theoretical analysis and empirical results on multiple tasks. In our experiments, SMoA improves average performance in the current lower-budget setting over LoRA and competitive LoRA-style baselines.

Problem

• Parameter-Efficient Fine-Tuning
• Low-Rank Adaptation
• Spectral Representation
• Model Compression
• Large Language Models

Innovation

• Spectrum Modulation
• Parameter-Efficient Fine-Tuning
• Low-Rank Adaptation
• Spectral Block Partitioning
• Hadamard Modulation