Problem
Research questions and friction points this paper is trying to address.
large language models
parameter-efficient fine-tuning
prompt tuning
hierarchical structures
HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts
🤖 AI Summary
为解决现有提示调优方法的局限性，提出HiVe框架，通过层次结构和垂直混合专家机制实现输入依赖的提示专业化，提高多任务学习性能。
📝 Abstract
As large language models (LLMs) continue to scale, parameter-efficient fine-tuning (PEFT) has become a practical alternative to full-parameter adaptation. Prompt tuning is effective, but existing approaches either use flat prompt structures or hierarchical structures with fixed prompt composition, limiting adaptive prompt specialization. To address this limitation, we propose HiVe, a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization. HiVe constructs a prompt hierarchy by leveraging inter-task relationships during training, and employs a vertical mixture-of-experts (V-MoE) mechanism at inference time to compose prompts up to the level of specialization required for each input. Experiments show that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Hierarchy-based Vertical Mixture-of-Experts
Input-dependent Specialization
Parameter-efficient Fine-tuning
🔎 Similar Papers
No similar papers found.
