LoRTA: Low Rank Tensor Adaptation of Large Language Models

2024-10-05

arXiv.org

AI Summary

To address the persistently high number of trainable parameters in efficient fine-tuning of large language models (LLMs), this paper proposes Low-Rank Tensor Adaptation (LoRTA). LoRTA is the first method to systematically integrate high-order CP decomposition into parameter-efficient fine-tuning (PEFT), unifying the modeling of structural redundancy both across and within layers—without manual tensorization or additional hyperparameters. It represents parameter updates via compact, flexible low-rank tensors. Extensive experiments across diverse tasks—including natural language understanding, instruction tuning, preference optimization, and protein folding—demonstrate its effectiveness. Compared to mainstream PEFT approaches such as LoRA, LoRTA reduces trainable parameters by up to 67% while maintaining comparable or superior performance. This achieves a unified improvement in both parameter efficiency and generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low Rank Adaptation (LoRA) is a popular Parameter Efficient Fine Tuning (PEFT) method that effectively adapts large pre-trained models for downstream tasks. LoRA parameterizes model updates using low-rank matrices at each layer, significantly reducing the number of trainable parameters and, consequently, resource requirements during fine-tuning. However, the lower bound on the number of trainable parameters remains high due to the use of the low-rank matrix model. Recent works have addressed this limitation by proposing low rank tensor parameterizations for model updates. However, they only exploit redundancy across layers, or tensorize individual matrices using ad-hoc schemes that introduce additional hyperparameters. In this work, we propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation compared to existing matrix and tensor based PEFT methods. Our experiments on Natural Language Understanding, Instruction Tuning, Preference Optimization and Protein Folding benchmarks demonstrate that our method can achieve a reduction in the number of parameters while maintaining comparable performance.

Problem

• Parameter Efficiency
• Pretrained Models
• Task Adaptation

Innovation

• CP Decomposition
• Parameter Reduction
• Natural Language Understanding