Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation

2025-06-19

AI Summary

Standard LoRA independently optimizes low-rank matrices, limiting representational capacity and generalization; applying conventional tensor-train (TT) decomposition separately to each LoRA matrix fails to improve parameter efficiency or performance. This paper proposes TensorGuide—a TT-guided joint adaptation framework that co-generates multiple correlated LoRA matrices via a unified TT structure, significantly enhancing expressivity and generalization without increasing trainable parameters. Its core innovations include the first-ever joint TT parameterization mechanism and structured generation driven by controllable Gaussian noise, with theoretical guarantees of superior optimization dynamics and reduced generalization error. Evaluated on quantum dot classification and GPT-2 fine-tuning, TensorGuide achieves higher accuracy and stronger scalability with fewer parameters, consistently outperforming both standard LoRA and TT-LoRA.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Learning to Search
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Low-Rank Adaptation (LoRA) is widely recognized for its parameter-efficient fine-tuning of large-scale neural models. However, standard LoRA independently optimizes low-rank matrices, which inherently limits its expressivity and generalization capabilities. While classical tensor-train (TT) decomposition can be separately employed on individual LoRA matrices, this work demonstrates that the classical TT-based approach neither significantly improves parameter efficiency nor achieves substantial performance gains. This paper proposes TensorGuide, a novel tensor-train-guided adaptation framework to overcome these limitations. TensorGuide generates two correlated low-rank LoRA matrices through a unified TT structure driven by controlled Gaussian noise. The resulting joint TT representation inherently provides structured, low-rank adaptations, significantly enhancing expressivity, generalization, and parameter efficiency without increasing the number of trainable parameters. Theoretically, we justify these improvements through neural tangent kernel analyses, demonstrating superior optimization dynamics and enhanced generalization. Extensive experiments on quantum dot classification and GPT-2 fine-tuning benchmarks demonstrate that TensorGuide-based LoRA consistently outperforms standard LoRA and TT-LoRA, achieving improved accuracy and scalability with fewer parameters.

Problem

• Enhance expressivity and generalization in Low-Rank Adaptation
• Improve parameter efficiency without increasing trainable parameters
• Overcome limitations of standard LoRA and TT-LoRA approaches

Innovation

• Joint TT structure for correlated LoRA matrices
• Controlled Gaussian noise drives unified TT
• Enhanced expressivity without extra parameters