tCURLoRA: Tensor CUR Decomposition Based Low-Rank Parameter Adaptation for Medical Image Segmentation

2025-01-04

AI Summary

To address the computational and memory bottlenecks of full fine-tuning large models for medical image segmentation under resource-constrained settings, this paper proposes a lightweight parameter-efficient fine-tuning (PEFT) method based on tensor CUR decomposition. It is the first to introduce tensor CUR decomposition into PEFT, overcoming the limitations of conventional matrix decomposition in modeling high-dimensional weight structures. By reconstructing weights as 3D tensors and updating only low-rank components, the method more accurately captures higher-order correlations among model parameters. The approach integrates tensorized weight concatenation, an extended LoRA architecture, and a low-rank adaptation mechanism. Evaluated on multiple medical image segmentation benchmarks, it achieves over 98% of full fine-tuning performance while introducing fewer than 0.1% trainable parameters—significantly outperforming mainstream PEFT methods such as LoRA.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Transfer learning, by leveraging knowledge from pre-trained models, has significantly enhanced the performance of target tasks. However, as deep neural networks scale up, full fine-tuning introduces substantial computational and storage challenges in resource-constrained environments, limiting its widespread adoption. To address this, parameter-efficient fine-tuning (PEFT) methods have been developed to reduce computational complexity and storage requirements by minimizing the number of updated parameters. While matrix decomposition-based PEFT methods, such as LoRA, show promise, they struggle to fully capture the high-dimensional structural characteristics of model weights. In contrast, high-dimensional tensors offer a more natural representation of neural network weights, allowing for a more comprehensive capture of higher-order features and multi-dimensional interactions. In this paper, we propose tCURLoRA, a novel fine-tuning method based on tensor CUR decomposition. By concatenating pre-trained weight matrices into a three-dimensional tensor and applying tensor CUR decomposition, we update only the lower-order tensor components during fine-tuning, effectively reducing computational and storage overhead. Experimental results demonstrate that tCURLoRA outperforms existing PEFT methods in medical image segmentation tasks.

Problem

• Resource-limited Environment
• Fine-tuning Large Pre-trained Models
• Medical Image Processing

Innovation

• Tensor Decomposition
• Pre-trained Model Fine-tuning
• Medical Image Processing