ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion

2025-03-31

AI Summary

Existing LoRA adaptation methods struggle to balance scalability with fine-grained controllability. To address this, we propose ORAL, the first framework leveraging a conditional recurrent diffusion model to directly generate task-specific LoRA parameters—eliminating the need for fine-tuning or retraining. ORAL jointly models the base model architecture and textual/multimodal task prompts, enabling cross-model parameter transfer via recurrent generation while maintaining efficiency and precise control even at billion-parameter scales. Extensive experiments across 7 languages, 4 vision tasks, 3 multimodal tasks, and 5 pre-trained LLMs demonstrate that ORAL-generated LoRA adapters match or surpass full fine-tuning in performance, while drastically reducing adaptation cost. This work establishes the first systematic application of diffusion-based generative paradigms to lightweight adaptation of large language and multimodal models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Parameter generation has emerged as a novel paradigm for neural network development, offering an alternative to traditional neural network training by synthesizing high-quality model weights directly. In the context of Low-Rank Adaptation (LoRA) for evolving ($ extit{i.e.}$, constantly updated) large language models (LLMs), this approach promises efficient adaptation without costly retraining. However, existing methods face critical limitations in simultaneously achieving scalability and controllability. In this paper, we introduce $ exttt{ORAL}$, a novel $ extbf{conditional recurrent diffusion}$ framework that addresses these challenges. $ exttt{ORAL}$ incorporates a novel conditioning mechanism that integrates model architecture and textual task specifications, enabling the generation of task-specific LoRA parameters that can seamlessly transfer across evolving foundation models. Our approach successfully scales to billions-of-parameter LLMs and maintains controllability. Through extensive experiments across seven language tasks, four vision tasks, and three multimodal tasks using five pre-trained LLMs, we demonstrate that $ exttt{ORAL}$ generates high-quality LoRA parameters that achieve comparable or superior performance to vanilla trained counterparts.

Problem

• Efficient adaptation of evolving large language models
• Scalable and controllable LoRA parameter generation
• Task-specific parameter transfer across foundation models

Innovation

• Conditional recurrent diffusion for LoRA
• Generates task-specific LoRA parameters
• Scales to billion-parameter LLMs