High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning

2026-01-12

AI Summary

This work proposes SMoA, a high-rank structured modulation adapter that overcomes the limitations of low-rank adaptation methods such as LoRA, which are constrained by their low-rank assumption and often underperform full fine-tuning in capturing task-specific features. By selectively amplifying or suppressing key features across multiple subspaces while keeping the pre-trained weights frozen, SMoA enables high-rank parameter updates with fewer trainable parameters, thereby breaking the low-rank bottleneck and significantly enhancing model representational capacity. Evaluated on ten downstream tasks, SMoA consistently outperforms LoRA and its variants, and ablation studies confirm the effectiveness and superiority of its structural design.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

As the number of model parameters increases, parameter-efficient fine-tuning (PEFT) has become the go-to choice for tailoring pre-trained large language models. Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning, which is widely used to reduce resource requirements. However, decreasing the rank encounters challenges with limited representational capacity when compared to full parameter fine-tuning. We present \textbf{SMoA}, a high-rank \textbf{S}tructured \textbf{MO}dulation \textbf{A}dapter that uses fewer trainable parameters while maintaining a higher rank, thereby improving the model's representational capacity and offering improved performance potential. The core idea is to freeze the original pretrained weights and selectively amplify or suppress important features of the original weights across multiple subspaces. The subspace mechanism provides an efficient way to increase the capacity and complexity of a model. We conduct both theoretical analyses and empirical studies on various tasks. Experiment results show that SMoA outperforms LoRA and its variants on 10 tasks, with extensive ablation studies validating its effectiveness.

Problem

• parameter-efficient fine-tuning
• low-rank adaptation
• representational capacity
• large language models
• model adaptation

Innovation

• parameter-efficient fine-tuning
• high-rank adaptation
• structured modulation
• subspace mechanism
• LoRA