SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning

2026-04-20

AI Summary

Existing MoE-LoRA approaches struggle to effectively align input semantics with expert capabilities due to imprecise routing and insufficient task adaptability, and they fail to dynamically respond to varying task complexity. To address these limitations, this work proposes SAMoRA, a novel framework that introduces a semantic-aware router to accurately match inputs with the most suitable experts and incorporates a task-adaptive scaling mechanism to dynamically modulate expert contributions. Furthermore, SAMoRA employs a tailored joint regularization objective to jointly optimize expert specialization and scaling efficacy. Experimental results demonstrate that SAMoRA significantly outperforms current state-of-the-art methods across multiple multitask benchmarks, exhibiting superior generalization and task adaptability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The combination of Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) has shown significant potential for enhancing the multi-task learning capabilities of Large Language Models. However, existing methods face two primary challenges: (1)Imprecise Routing in the current MoE-LoRA method fails to explicitly match input semantics with expert capabilities, leading to weak expert specialization. (2)Uniform weight fusion strategies struggle to provide adaptive update strengths, overlooking the varying complexity of different tasks. To address these limitations, we propose SAMoRA (Semantic-Aware Mixture of LoRA Experts), a novel parameter-efficient fine-tuning framework tailored for task-adaptive learning. Specifically, A Semantic-Aware Router is proposed to explicitly align textual semantics with the most suitable experts for precise routing. A Task-Adaptive Scaling mechanism is designed to regulate expert contributions based on specific task requirements dynamically. In addition, a novel regularization objective is proposed to jointly promote expert specialization and effective scaling. Extensive experiments on multiple multi-task benchmarks demonstrate that SAMoRA significantly outperforms the state-of-the-art methods and holds excellent task generalization capabilities. Code is available at https://github.com/boyan-code/SAMoRA

Problem

• Mixture-of-Experts
• Low-Rank Adaptation
• task-adaptive learning
• semantic routing
• expert specialization

Innovation

• Semantic-Aware Routing
• Mixture-of-Experts
• LoRA
• Task-Adaptive Scaling
• Parameter-Efficient Fine-Tuning