Low-Rank Ternary Adaptation for Fine-Tuning Transformers

2026-08-25

AI Summary

本文提出了一种低秩三值乘性适应方法，直接在三值权重上进行微调，解决了现有方法需反量化或仅更新量化参数的问题。

Abstract

Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to ternary weights. This design is parameter-efficient and expressive, preserves the ternary domain, and supports direct merging without dequantization. Experiments on six models across language and vision, including ternarized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate that our method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines. Code is available at https://github.com/alexmanoo/ternary_adaptation.

Problem

• Ternary Transformers
• Fine-Tuning
• Low-bit LoRA

Innovation

• ternary multiplicative adaptation
• low-rank Kronecker factorization
• parameter-efficient
• preserves ternary domain
• direct merging without dequantization