FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts

2025-10-09

AI Summary

LoRA suffers from parameter interference during efficient fine-tuning, while MoE-enhanced LoRA mitigates intra-task correlations but incurs routing overhead and fails to address inter-task interference in multi-task merging. To tackle this, we propose FlyLoRA—an implicit mixture-of-experts (MoE) framework inspired by the Drosophila olfactory circuit. FlyLoRA replaces explicit routing with a frozen, sparse random projection matrix that jointly realizes rank-level expert activation and down-projection. This design preserves ultra-low parameter count while significantly enhancing task disentanglement and cross-task generalization. Extensive experiments across four domains—general knowledge, scientific question answering, mathematical reasoning, and code generation—demonstrate that FlyLoRA consistently outperforms state-of-the-art LoRA variants and MoE-based adaptations. Results validate its superior parameter efficiency, robustness against parameter interference, and strong generalization capability across diverse downstream tasks.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Generation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for foundation models, but it suffers from parameter interference, resulting in suboptimal performance. Although Mixture-of-Experts (MoE)-based LoRA variants show promise in mitigating intra-task correlations in single-task instruction tuning, they introduce additional router parameters and remain ineffective in multi-task model merging where inter-task interference arises. Inspired by the fly olfactory circuit, we propose FlyLoRA, an implicit MoE-based LoRA variant that introduces: (1) rank-wise expert activation in the up-projection matrix, and (2) an implicit router that unifies expert routing and down-projection, where a frozen sparse random projection matrix replaces the traditional dense trainable version. This design resolves the trade-off between intra-task decorrelation and computational efficiency by eliminating the need for an explicit router, while inherently mitigating inter-task interference due to the orthogonality property of random matrices. Extensive experiments across four domains -- general knowledge understanding, scientific question answering, mathematical reasoning, and code generation -- demonstrate consistent performance improvements over existing methods. Beyond empirical gains, FlyLoRA highlights how biological structures can inspire innovations in AI technologies. Code is available at https://github.com/gfyddha/FlyLoRA.

Problem

• Addresses parameter interference in LoRA fine-tuning
• Mitigates intra-task and inter-task interference simultaneously
• Improves task decoupling without explicit router parameters

Innovation

• Implicit router eliminates explicit routing parameters
• Rank-wise expert activation in up-projection matrix
• Frozen sparse random projection replaces dense trainable matrix