Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

2026-06-30

AI Summary

This work addresses a critical limitation of conventional Low-Rank Adaptation (LoRA), which neglects the nonlinear gating mechanism inherent in self-gated Transformer feedforward networks, leading to suboptimal channel selection. To remedy this, the authors propose a nonlinear-aware LoRA variant that, for the first time, integrates the local effective homogeneity induced by self-gating activations with low-rank updates. Their approach dynamically refines the selection and temporal evolution of responsive channels through structured gating adjustments—without introducing additional inference overhead or auxiliary losses. Specifically, it employs derivative-based temporal importance masks and activation-specific step-size scaling rules to modulate gating-related LoRA updates. Experiments demonstrate consistent improvements over standard LoRA across language model fine-tuning and vision-language transfer tasks, matching or surpassing the performance of other state-of-the-art parameter-efficient fine-tuning methods.

Abstract

Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks. In gated FFNs, a low-rank residual can change not only projected features but also the nonlinear selection weights that determine which channels contribute to the output. We formalize this effect as selection misalignment and connect it to the local effective homogeneity of self-gated activations. This motivates a nonlinearity-aware principle for parameter-efficient fine-tuning: low-rank updates should allocate capacity to gate channels whose nonlinear states remain responsive and should shape the temporal evolution of selection. We propose NA-LoRA, a training-only method with two lightweight mechanisms: a derivative-based temporal-importance mask for gate-related LoRA updates and an activation-specific step-scaling rule when a meaningful coarse effective-homogeneity partition is available. NA-LoRA adds no auxiliary loss and incurs no inference-time overhead. Experiments on language-model fine-tuning and vision-language transfer benchmarks show that NA-LoRA consistently improves over vanilla LoRA and is competitive with or better than strong PEFT variants.

Problem

• Low-rank adaptation
• Self-gated networks
• Selection misalignment
• Nonlinearity-aware
• Parameter-efficient fine-tuning

Innovation

• Nonlinearity-Aware
• LoRA
• Gated FFN
• Selection Misalignment
• Parameter-Efficient Fine-Tuning