PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts

2026-05-13

AI Summary

Existing parameter-efficient fine-tuning methods struggle to jointly optimize continuous prompts and model adaptation in multi-task learning, leading to limited expressiveness and suboptimal resource efficiency. This work proposes a novel parameter-efficient multi-task learning framework that, for the first time, integrates neural architecture search into continuous prompt design and co-optimizes it with low-rank adaptation (LoRA), enabling efficient joint adjustment of prompt structures and model weights. Evaluated on standard benchmarks including GLUE and SuperGLUE, the proposed method achieves an average accuracy improvement of 6.67%, with gains as high as 10.75% on certain tasks, significantly outperforming state-of-the-art approaches such as MTL-LoRA and MultiLoRA.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) is widely used for adapting Large Language Models (LLMs) for various tasks. Recently, there has been an increasing demand for fine-tuning a single LLM for multiple tasks because it requires overall less data for fine-tuning thanks to the common features shared among tasks. More importantly, LLMs are resource demanding and deploying a single model for multiple tasks facilitates resource consolidation and consumes significantly less resources compared to deploying individual large model for each task. Existing PEFT methods like LoRA and Prefix Tuning are designed to adapt LLMs to a specific task. LoRA and its variation focus on aligning the model itself for tasks, overlooking the importance of prompt tuning in multi-task learning while Prefix Tuning only adopts a simple architecture to optimize prompts, which limits the adaption capabilities for multi-task. To enable efficient fine-tuning for multi-task learning, it is important to co-optimize prompt optimization and model adaptation. In this work, we propose a Parameter-Efficient Multi-task Learning (\PM), which employs a neural architecture engineering method for optimizing the continuous prompts while also performing low-rank adaption for model weights. We prototype PEML by creating an automated framework for optimizing the continuous prompts and adapting model weights. We evaluate PEML against state-of-the-arts multi-task learning methods MTL-LoRA, MultiLoRa, C-Poly, and MoE, on the GLUE, SuperGLUE, Massive Multitask Language Understanding, and commonsense reasoning benchmarks. The evaluation results present an average accuracy improvement of up to 6.67%, with individual tasks showing peak gains of up to 10.75%.

Problem

• Parameter-Efficient Fine-Tuning
• Multi-Task Learning
• Large Language Models
• Prompt Tuning
• Model Adaptation

Innovation

• Parameter-Efficient Fine-Tuning
• Multi-Task Learning
• Continuous Prompts
• Low-Rank Adaptation
• Neural Architecture Engineering