MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

2024-10-12

arXiv.org

AI Summary

In multi-task learning, standard LoRA suffers from task interference and confusion due to its rigid constraint that all tasks share a single low-rank intrinsic subspace. To address this, we propose Task-Adaptive LoRA (TA-LoRA), the first LoRA variant featuring task-adaptive low-rank branches. TA-LoRA introduces learnable task identifier embeddings and gated low-rank projection matrices to explicitly decouple task-specific subspaces from cross-task shared subspaces. This enables parameter-efficient, task-aware joint adaptation without increasing inference overhead. Evaluated across diverse benchmarks—including natural language understanding, commonsense reasoning, vision-language comprehension, and industrial ad relevance ranking—TA-LoRA consistently outperforms LoRA and its variants under equivalent or fewer trainable parameters, achieving state-of-the-art multi-task performance while preserving LoRA’s parameter efficiency.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Parameter-efficient fine-tuning (PEFT) has been widely employed for domain adaptation, with LoRA being one of the most prominent methods due to its simplicity and effectiveness. However, in multi-task learning (MTL) scenarios, LoRA tends to obscure the distinction between tasks by projecting sparse high-dimensional features from different tasks into the same dense low-dimensional intrinsic space. This leads to task interference and suboptimal performance for LoRA and its variants. To tackle this challenge, we propose MTL-LoRA, which retains the advantages of low-rank adaptation while significantly enhancing MTL capabilities. MTL-LoRA augments LoRA by incorporating additional task-adaptive parameters that differentiate task-specific information and capture shared knowledge across various tasks within low-dimensional spaces. This approach enables pre-trained models to jointly adapt to different target domains with a limited number of trainable parameters. Comprehensive experimental results, including evaluations on public academic benchmarks for natural language understanding, commonsense reasoning, and image-text understanding, as well as real-world industrial text Ads relevance datasets, demonstrate that MTL-LoRA outperforms LoRA and its various variants with comparable or even fewer learnable parameters in MTL setting.

Problem

• Addresses task interference in multi-task learning with LoRA
• Enhances differentiation of task-specific information in low-rank adaptation
• Improves performance across diverse domains with minimal trainable parameters

Innovation

• Augments LoRA with task-adaptive parameters
• Differentiates task-specific and shared knowledge
• Enhances multi-task learning with low-rank adaptation