LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection

2026-05-12

AI Summary

This work addresses a critical limitation in existing parameter-efficient fine-tuning (PEFT) methods that conflate subspace selection with transformation within orthogonal adaptation and neglect the systematic design of support sets. To resolve this, we propose LOFT, a framework that models orthogonal fine-tuning as multiplicative subspace rotation, explicitly decoupling subspace selection from transformation. LOFT further establishes task-aware support set selection as a core design dimension, leveraging first-order analysis of downstream task gradients to guide support set construction. The framework unifies diverse orthogonal variants—including coordinate, butterfly, Householder, and principal subspaces—under a single formulation. Experiments across language understanding, vision transfer, mathematical reasoning, and multilingual out-of-distribution adaptation demonstrate that gradient-informed support sets substantially improve the efficiency–performance trade-off under constraints on parameters, memory, and computation.

Abstract

Orthogonal parameter-efficient fine-tuning (PEFT) adapts pretrained weights through structure-preserving multiplicative transformations, but existing methods often conflate two distinct design choices: the subspace in which adaptation occurs and the transformation applied within that subspace. This paper introduces LOFT, a low-rank orthogonal fine-tuning framework that explicitly separates these two components. By viewing orthogonal adaptation as a multiplicative subspace rotation, LOFT provides a unified formulation that recovers representative orthogonal PEFT methods, including coordinate-, butterfly-, Householder-, and principal-subspace-based variants. More importantly, this perspective exposes support selection as a central design axis rather than a byproduct of a particular parameterization. We develop a first-order analysis showing that useful adaptation supports should be informed by the downstream training signal, motivating practical task-aware support selection strategies. Across language understanding, visual transfer, mathematical reasoning, and multilingual out-of-distribution adaptation, LOFT recovers principal-subspace orthogonal adaptation while gradient-informed supports improve the efficiency-performance trade-off under matched parameter, memory, and compute budgets. These results suggest that principled support selection is an important direction for improving orthogonal PEFT.

Problem

• orthogonal fine-tuning
• parameter-efficient fine-tuning
• support selection
• subspace adaptation
• task-aware

Innovation

• orthogonal fine-tuning
• parameter-efficient fine-tuning
• support selection
• low-rank adaptation
• task-aware optimization