LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models

2026-07-07

AI Summary

Existing low-rank adaptation methods, such as LoRA, flatten 4D convolutional kernels into 2D matrices, disregarding their intrinsic spatial-channel coupling structure and thereby disrupting spatial topology and discarding valuable pre-trained priors. To address this limitation, this work proposes LoCA, the first spatially aware low-rank adaptation framework tailored for convolutional layers. LoCA decouples channel and spatial dimensions, applying low-rank channel adaptation to enable effective cross-channel mixing while leveraging singular value decomposition (SVD) to fine-tune the spatial bases of pre-trained kernels with high precision. This approach preserves spatial priors, substantially reduces the number of trainable parameters, and achieves state-of-the-art performance across fine-grained classification, domain-generalized semantic segmentation, and generative tasks, all while mitigating catastrophic forgetting.

Abstract

Pre-trained Vision Foundation Models (VFMs) provide strong visual representations for diverse downstream tasks. The key challenge of VFM adaptation stems from the prohibitive costs of full fine-tuning and catastrophic forgetting. To address this, Low-Rank Adaptation (LoRA) has emerged as the prevailing paradigm for Parameter-Efficient Fine-Tuning (PEFT). However, LoRA is typically designed for transformer self-attention layers parameterized by 2D matrices. Since convolutional kernels inherently couple spatial and channel information within a 4D tensor, forcing them into a monolithic 2D matrix disrupts the inherent spatial topology. In this paper, we propose Low-Rank Convolutional Adaptation (LoCA), a convolution-aware PEFT framework that addresses spatial-channel entanglement by decoupling channel and spatial adaptation. LoCA introduces a low-rank channel adaptation for dense cross-channel mixing and refines spatial bases extracted from pre-trained kernels via Singular Value Decomposition (SVD). Experimental results show that LoCA preserves pre-trained spatial priors and achieves competitive or state-of-the-art performance across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks.

Problem

• Vision Foundation Models
• Parameter-Efficient Fine-Tuning
• Convolutional Adaptation
• Spatial-Channel Entanglement
• Low-Rank Adaptation

Innovation

• Low-Rank Adaptation
• Convolutional Adaptation
• Spatial-Channel Decoupling
• Parameter-Efficient Fine-Tuning
• Singular Value Decomposition