Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing
This work addresses the limitations of existing parameter-efficient fine-tuning methods, such as LoRA, which rely on fixed architectures and struggle in dynamic scenarios involving task conflicts or sensor failures. The authors propose a novel low-rank mixture-of-experts architecture that integrates local spatial partitioning with context-aware dynamic routing, available in both block-level and unit-level variants. A key innovation is a decentralized unit-level gating mechanism that achieves performance approaching ideal global routing without requiring centralized coordination. By combining block-wise low-rank decomposition, dynamically sparse routing, and a matrix-grid coordinate field model, the method establishes gradient firewalls to suppress error propagation. Experiments on high-dimensional matrix simulation, tabular data transformation, and vision tasks under sensor degradation demonstrate substantial improvements over static baselines, achieving robust and efficient dynamic adaptation.