LoCO: Low-rank Compositional Rotation Fine-tuning

2026-05-15

AI Summary

Existing parameter-efficient fine-tuning methods struggle to preserve the geometric structure of pre-trained model representations under low-rank updates. This work proposes a novel fine-tuning approach that constructs orthogonal transformations using low-rank skew-symmetric matrices and introduces parallelizable composition chains of rotations. By doing so, it simultaneously ensures orthogonality, computational efficiency, and controllable approximation error in high-dimensional spaces. The method uniquely integrates low-rank compositional rotations with orthogonal constraints, achieving state-of-the-art or competitive performance across diverse tasks—including diffusion Transformers, vision Transformers, and language model fine-tuning—outperforming both existing orthogonal and non-orthogonal approaches.

Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as an critical technique for adapting large-scale foundation models across natural language processing and computer vision. While existing methods such as low-rank adaptations achieve parameter efficiency via low-rank weight updates, they are limited in their ability to preserve the geometric structure of pretrained representations. We introduce Low-rank Compositional Orthogonal fine-tuning (LoCO), a novel PEFT method that constructs orthogonal transformations through low-rank skew-symmetric matrices and compositional rotation chains. We propose an approximation scheme that enables fully parallel computation of compositional rotations, making the approach practical for high-dimensional feature spaces. Our method maintains low computational complexity while maintaining orthogonality with controlled approximation error. We validate LoCO across diverse domains, including diffusion transformer fine-tuning, vision transformer adaptation, and language model adaptation. Our method demonstrates superior or competitive performance compared to both existing orthogonal and non-orthogonal methods.

Problem

• parameter-efficient fine-tuning
• geometric structure preservation
• orthogonal transformations
• low-rank adaptation
• foundation models

Innovation

• parameter-efficient fine-tuning
• orthogonal transformation
• low-rank adaptation
• compositional rotation
• skew-symmetric matrix