Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space

2026-08-31

AI Summary

研究通过在量化码空间中使用梯度来微调低位模型，以解决低位模型适应性优化问题，提出代码替代梯度方法加速优化并保持部署一致性。

Abstract

Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practically important as it reduces memory and inference cost for storage and deployment. Under this constraint, adaptation becomes an optimization problem over quantization codes and scales. Existing continuous low-bit training is efficient, but it can be distorted by straight through estimation error or by post-quantize gap; discrete search is deployment-faithful, but it is often too inefficient under a finite training budget. We propose code surrogate gradient as the first order signal in deployable code space to acceleate optimization, and performing guided search to preserve deployment faithfulness. Experiments across arithmetic reasoning, instruction following, and structured language understanding show that GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes. Code is provided at https://github.com/ovo67/GradCodes.

Problem

• low-bit models
• quantization
• fine-tuning
• deployment
• optimization

Innovation

• code surrogate gradient
• low-bit models
• quantization
• guided search
• deployment faithfulness