TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Models

2025-04-25

AI Summary

To address the challenge of balancing parameter efficiency and representational capacity in efficient fine-tuning of large language models (LLMs), this paper proposes Tri-Matrix Low-Rank Adaptation (TMLoRA). TMLoRA decomposes weight updates into two fixed Gaussian random matrices, one trainable low-rank matrix, and layer-wise adaptive learnable scaling factors. This design achieves performance on par with LoRA on the GLUE benchmark while using significantly fewer trainable parameters—setting a new state-of-the-art in parameter efficiency. Theoretical and empirical analyses reveal that TMLoRA induces Gaussian-like weight distributions, ensures norm stability, and enables heterogeneous layer-wise scaling—jointly enhancing expressivity and interpretability. Further analyses of feature-space dynamics, spectral distributions, and update directions confirm its strong alignment with LoRA’s behavior. Collectively, TMLoRA establishes a novel paradigm for efficient LLM adaptation, bridging high fidelity and extreme parameter sparsity.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We propose TLoRA, a novel tri-matrix low-rank adaptation method that decomposes weight updates into three matrices: two fixed random matrices and one trainable matrix, combined with a learnable, layer-wise scaling factor. This tri-matrix design enables TLoRA to achieve highly efficient parameter adaptation while introducing minimal additional computational overhead. Through extensive experiments on the GLUE benchmark, we demonstrate that TLoRA achieves comparable performance to existing low-rank methods such as LoRA and Adapter-based techniques, while requiring significantly fewer trainable parameters. Analyzing the adaptation dynamics, we observe that TLoRA exhibits Gaussian-like weight distributions, stable parameter norms, and scaling factor variability across layers, further highlighting its expressive power and adaptability. Additionally, we show that TLoRA closely resembles LoRA in its eigenvalue distributions, parameter norms, and cosine similarity of updates, underscoring its ability to effectively approximate LoRA's adaptation behavior. Our results establish TLoRA as a highly efficient and effective fine-tuning method for LLMs, offering a significant step forward in resource-efficient model adaptation.

Problem

• Efficient parameter adaptation for large language models
• Minimal computational overhead in model fine-tuning
• Comparable performance with fewer trainable parameters

Innovation

• Tri-matrix low-rank adaptation design
• Fixed random and trainable matrices combination
• Layer-wise learnable scaling factor integration