In-Context Meta LoRA Generation

2025-01-29

AI Summary

To address parameter redundancy and cross-task knowledge fragmentation in multi-task LoRA adapters, this paper proposes a Contextual Meta-Learning-based Conditional Variational Autoencoder (CVAE) framework. It is the first to integrate in-context meta-learning with CVAE, enabling zero-shot generation of high-fidelity, task-specific LoRA weights solely from natural-language task descriptions. The method explicitly models the mapping from task semantics to parameter distributions, facilitating cross-task knowledge transfer and precise parameter reconstruction. Under multi-task joint training, our approach achieves significantly higher LoRA weight reconstruction accuracy than state-of-the-art methods. The resulting model occupies only 283 MB—approximately 1% of the total storage required by conventional LoRA adapters—while supporting plug-and-play deployment of task-specialized large language models without any fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-rank Adaptation (LoRA) has demonstrated remarkable capabilities for task specific fine-tuning. However, in scenarios that involve multiple tasks, training a separate LoRA model for each one results in considerable inefficiency in terms of storage and inference. Moreover, existing parameter generation methods fail to capture the correlations among these tasks, making multi-task LoRA parameter generation challenging. To address these limitations, we propose In-Context Meta LoRA (ICM-LoRA), a novel approach that efficiently achieves task-specific customization of large language models (LLMs). Specifically, we use training data from all tasks to train a tailored generator, Conditional Variational Autoencoder (CVAE). CVAE takes task descriptions as inputs and produces task-aware LoRA weights as outputs. These LoRA weights are then merged with LLMs to create task-specialized models without the need for additional fine-tuning. Furthermore, we utilize in-context meta-learning for knowledge enhancement and task mapping, to capture the relationship between tasks and parameter distributions. As a result, our method achieves more accurate LoRA parameter generation for diverse tasks using CVAE. ICM-LoRA enables more accurate LoRA parameter reconstruction than current parameter reconstruction methods and is useful for implementing task-specific enhancements of LoRA parameters. At the same time, our method occupies 283MB, only 1% storage compared with the original LoRA.

Problem

• Multi-task Learning
• LoRA Models
• Parameter Efficiency

Innovation

• ICM-LoRA
• CVAE
• Meta-Learning