Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks

2026-05-03

AI Summary

This work proposes Flexi-LoRA, a novel framework that addresses the limitations of existing static low-rank adaptation (LoRA) methods, which suffer from fixed rank assignments that cannot adapt to varying input complexity, thereby constraining both efficiency and performance. Flexi-LoRA introduces, for the first time, a consistent input-adaptive rank allocation mechanism that operates uniformly during both training and inference. By dynamically adjusting the LoRA rank based on input complexity and integrating adaptive fine-tuning strategies across diverse tasks—including question answering, mathematical reasoning, and speech processing—the method achieves superior performance over static LoRA while significantly reducing parameter usage. Notably, Flexi-LoRA demonstrates pronounced gains on tasks requiring rigorous reasoning chains, effectively validating the efficacy and generalizability of dynamic rank allocation in parameter-efficient fine-tuning.

Abstract

Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become essential for deploying large language models, yet their static parameter allocation remains suboptimal for inputs of varying complexity. We present Flexi-LoRA, a novel framework that dynamically adjusts LoRA ranks based on input complexity during both training and inference. Through empirical analysis across question answering, mathematical reasoning, and speech tasks, we demonstrate that maintaining consistency between training and inference dynamics is important for effective adaptation, particularly for sequential reasoning tasks. Our findings reveal that input-dependent parameter allocation achieves higher performance with fewer parameters by optimally matching rank configurations to question complexity. Furthermore, task-specific dependency on rank dynamics varies, with mathematical reasoning tasks exhibiting higher dependency than QA tasks. Successful adaptation manifests not only in correctness but also in reasoning quality and instruction adherence. Flexi-LoRA consistently outperforms static LoRA while using fewer parameters, with performance gains more pronounced on tasks requiring strict reasoning chains. Our approach realizes key benefits of mixture-of-experts frameworks through a more streamlined implementation, reducing parameter redundancy while improving model capabilities. We provide comprehensive empirical studies across diverse tasks, establishing a basis for future work in input-adaptive and efficient fine-tuning approaches.

Problem

• parameter-efficient fine-tuning
• input-adaptive ranks
• Low-Rank Adaptation
• dynamic rank allocation
• model efficiency

Innovation

• Flexi-LoRA
• input-adaptive ranks
• parameter-efficient fine-tuning
• dynamic rank allocation
• reasoning quality