Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights

2025-06-19

AI Summary

To address the limitation of conventional PEFT methods (e.g., LoRA), which require task-specific fine-tuning for each downstream task, this work proposes a prompt-conditioned parameter generation paradigm: given only a few unlabeled task prompts, it enables zero-shot, gradient-free, sub-second generation of corresponding LoRA adapter weights. The method employs a lightweight text encoder coupled with a cascaded hyperconvolutional decoder to map prompt embeddings end-to-end to full LoRA weight matrices. This framework is the first to achieve cross-task and cross-domain zero-shot adapter weight generation—without any labels or optimization iterations. Compared to full-parameter fine-tuning, it reduces GPU memory and computational overhead by 12,000×. On benchmarks spanning commonsense reasoning, mathematical problem solving, code generation, and multimodal understanding, it achieves an average 30% performance gain over the best-performing LoRA variants.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce the cost of customizing large language models (LLMs), yet still require a separate optimization run for every downstream dataset. We introduce extbf{Drag-and-Drop LLMs ( extit{DnD})}, a prompt-conditioned parameter generator that eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates. A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices. Once trained in a diverse collection of prompt-checkpoint pairs, DnD produces task-specific parameters in seconds, yielding i) up to extbf{12,000$ imes$} lower overhead than full fine-tuning, ii) average gains up to extbf{30%} in performance over the strongest training LoRAs on unseen common-sense reasoning, math, coding, and multimodal benchmarks, and iii) robust cross-domain generalization despite never seeing the target data or labels. Our results demonstrate that prompt-conditioned parameter generation is a viable alternative to gradient-based adaptation for rapidly specializing LLMs. Our project is available at href{https://jerryliang24.github.io/DnD}{https://jerryliang24.github.io/DnD}.

Problem

• Eliminates per-task training for LLMs with prompt-to-weights mapping
• Reduces overhead by up to 12,000x compared to fine-tuning
• Improves performance on unseen tasks by 30% on average

Innovation

• Prompt-conditioned parameter generator for LoRA weights
• Lightweight text encoder for prompt embeddings
• Hyper-convolutional decoder transforms embeddings to LoRA matrices