Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations

2025-09-26

AI Summary

The linear nature of Low-Rank Adaptation (LoRA) limits its representational capacity in parameter-efficient fine-tuning of large language models. To address this, we propose LoRAN—a parameter-preserving nonlinear extension of LoRA—that introduces structured lightweight transformations incorporating a novel sinusoidal activation function, Sinter, thereby injecting strong nonlinearity while maintaining the low-rank structure. Sinter is deliberately designed to balance periodic representation capability and gradient stability, outperforming standard activations (e.g., Sigmoid, ReLU, Tanh) in both theoretical properties and empirical behavior. Extensive experiments demonstrate that LoRAN consistently surpasses QLoRA and other LoRA variants on summarization and text classification benchmarks. Ablation studies confirm that performance gains stem specifically from Sinter’s structural nonlinearity enhancement—not from increased parameter count—validating LoRAN’s efficacy as a principled, parameter-efficient nonlinear adaptation framework.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN, a non-linear extension of LoRA that applies lightweight transformations to the low-rank updates. We further introduce Sinter, a sine-based activation that adds structured perturbations without increasing parameter count. Experiments across summarization and classification tasks show that LoRAN consistently improves over QLoRA. Ablation studies reveal that Sinter outperforms standard activations such as Sigmoid, ReLU, and Tanh, highlighting the importance of activation design in lowrank tuning.

Problem

• Extending LoRA with nonlinear transformations for better expressiveness
• Introducing sine-based activation to enhance performance without extra parameters
• Improving fine-tuning efficiency across summarization and classification tasks

Innovation

• LoRAN extends LoRA with lightweight nonlinear transformations
• Sinter uses sine-based activation for structured perturbations
• Method improves performance without increasing parameter count