Strategic Over-Parameterization for Generalizable Low-Rank Adaptation

2026-05-15

AI Summary

This work addresses the limited generalization of existing parameter-efficient fine-tuning methods—such as LoRA—across tasks and domains, which often struggle to balance computational efficiency with adaptation performance. The authors propose LoRA-Over, a framework that introduces a collapsible over-parameterized auxiliary structure to expand the optimization landscape while enabling lossless compression back to the standard low-rank form during inference, thereby incurring zero additional overhead. By integrating both static and dynamic capacity scheduling strategies, LoRA-Over consistently outperforms standard LoRA across multiple benchmarks—including GLUE, MT-Bench, GSM8K, and HumanEval—using LLaMA 2-7B and LLaMA 3.1-8B models, achieving a synergistic improvement in both parameter efficiency and generalization capability.

Abstract

Adapting large language models (LLMs) to downstream tasks via full fine-tuning is increasingly impractical due to its computational and memory demands. Parameter-efficient fine-tuning (PEFT) approaches such as Low-Rank Adaptation (LoRA) mitigate this by confining updates to a compact set of trainable parameters, but this aggressive reduction often sacrifices generalization, especially under transfer across heterogeneous tasks and domains. We revisit the tension between parameter efficiency and adaptation capacity, and ask whether the two are truly at odds. We answer in the negative by introducing LoRA-Over, a framework grounded in a simple principle: enrich the optimization landscape during training, then collapse the enrichment at inference. LoRA-Over injects auxiliary parameters into the low-rank adapters during training to broaden the effective hypothesis space, and through a decomposition-based reformulation folds them back into a standard low-rank structure with negligible reconstruction error, keeping inference cost identical to vanilla LoRA. Since not all weight matrices benefit equally from added capacity, we further propose two scheduling strategies, one statically predefined and one dynamically determined at runtime, that direct extra capacity where most needed. We evaluate LoRA-Over on language understanding (GLUE, T5-Base), dialogue (MT-Bench), arithmetic reasoning (GSM8K), and code generation (HumanEval), using LLaMA 2-7B and LLaMA 3.1-8B. Across all benchmarks and scales, LoRA-Over consistently outperforms vanilla LoRA, showing that principled over-parameterization designed to vanish at inference is an effective lever for improving PEFT generalization. Code will be released upon acceptance.

Problem

• parameter-efficient fine-tuning
• Low-Rank Adaptation
• generalization
• over-parameterization
• large language models

Innovation

• Low-Rank Adaptation
• Over-Parameterization
• Parameter-Efficient Fine-Tuning
• Generalization
• Inference-Efficient Training