Text-to-LoRA: Instant Transformer Adaption

2025-06-06

AI Summary

Traditional foundation models require extensive labeled data and iterative fine-tuning to adapt to new tasks, incurring high computational costs and sensitivity to hyperparameters. To address this, we propose Text-to-LoRA (T2L), the first method enabling zero-shot generation of task-specific LoRA adapters directly from natural language task descriptions—without gradient-based optimization. T2L introduces a text-driven LoRA generation paradigm that integrates hypernetworks, instruction encoding, multi-task meta-training, and low-rank adaptation. It synthesizes adapters via a single forward pass, compressing hundreds of LoRA modules into a unified hypernetwork. Evaluated on nine benchmarks—including GSM8K and ARC—T2L achieves performance comparable to task-specific fine-tuned LoRAs, while drastically reducing deployment overhead and adaptation latency.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

While Foundation Models provide a general tool for rapid content creation, they regularly require task-specific adaptation. Traditionally, this exercise involves careful curation of datasets and repeated fine-tuning of the underlying model. Fine-tuning techniques enable practitioners to adapt foundation models for many new applications but require expensive and lengthy training while being notably sensitive to hyper-parameter choices. To overcome these limitations, we introduce Text-to-LoRA (T2L), a model capable of adapting Large Language Models on the fly solely based on a natural language description of the target task. T2L is a hypernetwork trained to construct LoRAs in a single inexpensive forward pass. After training T2L on a suite of 9 pre-trained LoRA adapters (GSM8K, Arc, etc.), we show that the ad-hoc reconstructed LoRA instances match the performance of task-specific adapters across the corresponding test sets. Furthermore, T2L can compress hundreds of LoRA instances and zero-shot generalize to entirely unseen tasks. This approach provides a significant step towards democratizing the specialization of foundation models and enables language-based adaptation with minimal compute requirements. Our code is available at https://github.com/SakanaAI/text-to-lora

Problem

• Adapting foundation models without lengthy fine-tuning
• Reducing sensitivity to hyper-parameter choices in adaptation
• Enabling task-specific adaptation via natural language descriptions

Innovation

• Text-to-LoRA enables instant LLM adaptation via text
• Hypernetwork generates LoRAs in one forward pass
• Compresses and generalizes LoRAs for unseen tasks