LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

2025-02-12

AI Summary

To address the high GPU memory overhead and severe accuracy degradation under ultra-low-bit quantization (<2 bits) in LoRA fine-tuning of large language models (LLMs), this paper proposes the first framework enabling end-to-end 1.15-bit LoRA training. Methodologically, we introduce a fine-grained quantization mapping, dynamic threshold selection, and adaptive bit-width allocation, accelerated by custom CUDA kernels for efficient training. Our key contribution is breaking the sub-2-bit accuracy bottleneck in LoRA, achieving the first practical ultra-low-bit LoRA fine-tuning without significant performance loss. Extensive experiments across four mainstream LLMs and four benchmark datasets demonstrate that our method reduces GPU memory consumption by up to 50% compared to standard LoRA, while incurring an average accuracy drop of less than 1% at 1.15 bits—substantially improving training efficiency and scalability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss. LowRA optimizes fine-grained quantization - mapping, threshold selection, and precision assignment - while leveraging efficient CUDA kernels for scalable deployment. Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance-precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50%. Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments.

Problem

• Reduces LoRA fine-tuning resource costs
• Enables sub-2-bit parameter fine-tuning
• Optimizes quantization for minimal performance loss

Innovation

• LoRA fine-tuning under 2 bits
• Fine-grained quantization optimization
• Efficient CUDA kernels deployment