Effective LoRA Adapter Routing using Task Representations

2026-01-29

AI Summary

This work proposes LORAUTER, a novel framework addressing the challenge of efficient selection and composition of adapters from large-scale LoRA pools. LORAUTER introduces, for the first time, a task-level routing mechanism that operates without requiring adapter-specific training data. By generating task embeddings from a small validation set, routing decisions are made based on task representations rather than adapter features, ensuring computational overhead scales only with the number of tasks. Integrating low-rank adaptation, task representation learning, and adapter composition strategies, LORAUTER achieves 101.2% of oracle performance in multi-task settings, outperforms baselines by 5.2 points on unseen tasks, and demonstrates robustness in pools containing over 1,500 noisy adapters, significantly enhancing generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Low-rank adaptation (LoRA) enables parameter efficient specialization of large language models (LLMs) through modular adapters, resulting in rapidly growing public adapter pools spanning diverse tasks. Effectively using these adapters requires routing: selecting and composing the appropriate adapters for a query. We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics. Unlike existing approaches that map queries directly to adapters, LORAUTER routes queries via task embeddings derived from small validation sets and does not require adapter training data. By operating at the task level, LORAUTER achieves efficient routing that scales with the number of tasks rather than the number of adapters. Experiments across multiple tasks show that LORAUTER consistently outperforms baseline routing approaches, matching Oracle performance (101.2%) when task-aligned adapters exist and achieving state-of-the-art results on unseen tasks (+5.2 points). We further demonstrate the robustness of LORAUTER to very large, noisy adapter pools by scaling it to over 1500 adapters.

Problem

• LoRA
• adapter routing
• task representation
• large language models
• parameter-efficient adaptation

Innovation

• LoRA
• adapter routing
• task representation
• parameter-efficient fine-tuning
• large language models