🤖 AI Summary
This work addresses the challenges of parameter incompatibility and unreliable compositional outputs when integrating LoRA adapters retrieved from an open pool. To tackle these issues, the authors propose the SCALE framework, which employs a Layer-Adaptive Sparse Residual Composition (LASRC) mechanism to perform residual merging while preserving critical linear anchors. SCALE further introduces multi-view disagreement analysis as an uncertainty signal and leverages a support-set loss proxy to evaluate reliability and guide adapter selection. Experimental results demonstrate that SCALE significantly enhances single-view performance on benchmarks such as FLAN-T5-Large and BBH, confirming its effectiveness and generalization capability across diverse decoder architectures.
📝 Abstract
Libraries of Low-Rank Adaptation (LoRA) adapters are becoming a practical by-product of parameter-efficient adaptation. Once such adapters accumulate, a natural question is no longer how to train one adapter for one task, but how to reuse an open pool of adapters for a new task given only a small support set. Prior work has shown that LoRA modules can be composed at the task level and dynamically selected at the instance level. However, open-pool LoRA reuse is not automatic: retrieving relevant adapters does not guarantee that their parameter updates are compatible, and composing adapters does not guarantee reliable outputs.
We introduce the Sparse-Composition Agreement Layer (SCALE), a post-retrieval audit and composition framework for open-pool LoRA reuse. SCALE contains a deployable 1.0* merge path, Layer-Adaptive Sparse Residual Composition (LASRC), and a higher-cost reliability-analysis layer for multi-view disagreement. LASRC addresses merge interference by preserving a linear anchor while residualizing block-wise adapter update directions. The reliability layer treats disagreement among sparse composition views as an observable uncertainty signal and compares agreement, support-loss proxy selection, and oracle headroom under explicit path cost. In matched FLAN-T5-Large, BIG-Bench Hard (BBH), and 97-LoRA experiments, LASRC gives a directional single-view gain under fixed retrieval, while SCALE-support is reported as a query-label-free 3.0* reliability-analysis variant rather than as a calibrated or throughput-equivalent selector. Protocol-distinct BBH-8 validation shows the same qualitative trend on three decoder-only backbones. Detailed scores, paired audits, and path-cost records are reported in the experimental section.