HOFT: Householder Orthogonal Fine-tuning

2025-05-22

AI Summary

Orthogonal fine-tuning methods exhibit strong generalization but suffer from high computational and memory overhead. To address this, we propose HOFT, a Householder-transform-based orthogonal fine-tuning framework, and its scalable variant SHOFT. HOFT pioneers a novel paradigm that parameterizes orthogonal updates via Householder matrices; we theoretically establish its convergence properties and scaling behavior, and further integrate low-rank approximation with gradient-constrained optimization. Empirically, HOFT and SHOFT match or surpass state-of-the-art methods—including LoRA and QLoRA—on diverse tasks: commonsense reasoning, machine translation, topic generation, and mathematical reasoning. They reduce training memory consumption by 37% and inference latency by 29%. Our core contribution is the first efficient orthogonal fine-tuning paradigm that simultaneously achieves strong generalization and substantial efficiency gains—advancing both theoretical understanding and practical deployment of orthogonal adaptation in large language models.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Adaptation of foundation models using low-rank methods is a widespread approach. Another way to adapt these models is to employ orthogonal fine-tuning methods, which are less time and memory efficient despite their good generalization properties. In this work, we propose Householder Orthogonal Fine-tuning (HOFT), a novel orthogonal fine-tuning method that aims to alleviate time and space complexity. Moreover, some theoretical properties of the orthogonal fine-tuning paradigm are explored. From this exploration, Scaled Householder Orthogonal Fine-tuning (SHOFT) is proposed. Both HOFT and SHOFT are evaluated in downstream tasks, namely commonsense reasoning, machine translation, subject-driven generation and mathematical reasoning. Compared with state-of-the-art adaptation methods, HOFT and SHOFT show comparable or better results.

Problem

• Proposes HOFT for efficient orthogonal fine-tuning of foundation models
• Explores theoretical properties of orthogonal fine-tuning paradigm
• Evaluates HOFT and SHOFT in diverse downstream tasks

Innovation

• Householder Orthogonal Fine-tuning (HOFT) method
• Scaled Householder Orthogonal Fine-tuning (SHOFT)
• Reduces time and space complexity