Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer

2026-05-27

AI Summary

This work addresses the performance degradation of task vectors when transferred across differently parameterized pretrained models by proposing BiCo, a training-free dual-space coordinate alignment framework. BiCo is the first to interpret task vector formation through the lens of bilinear interaction, framing transfer as a joint alignment problem between input activation and output gradient spaces. It resolves this via a single forward–backward pass to compute two orthogonal Procrustes mappings. Without requiring any fine-tuning, BiCo consistently outperforms existing methods across diverse cross-model settings—spanning variations in model width, depth, and pretraining configurations—and achieves performance close to direct fine-tuning on multiple vision and language benchmarks.

Abstract

Fine-tuning large-scale pre-trained models is a recent prevalent paradigm for adapting general representations to specialized tasks. However, when a new version of a pre-trained model becomes available, expertise acquired through fine-tuning cannot be directly reused because it is tied to the parameterization of the original model, requiring another costly fine-tuning. To address this inefficiency, recent work uses task vectors, defined as the parameter difference between a fine-tuned model and its base model, to transfer expertise across models. While existing methods bridge disparate models by matching activations or gradients, a significant performance gap remains relative to direct fine-tuning, suggesting that these partial correspondences are insufficient. In this work, instead of viewing a task vector merely as a parameter offset, we revisit the formation of task vectors and show that they can be derived as accumulated bilinear interactions between input-side activations and output-side gradients. Motivated by this observation, we formulate task-vector transfer as a dual-space alignment problem and propose BiCo, a training-free framework for transferring task vectors through Bilinear Coordinate alignment. BiCo estimates orthogonal Procrustes mappings in both spaces using a single forward-backward pass on a small calibration set, without any parameter update. Across extensive computer vision and natural language processing benchmarks, BiCo consistently outperforms existing transfer methods across models that differ in width, depth, and pre-training configuration.

Problem

• task-vector transfer
• model adaptation
• parameter alignment
• fine-tuning reuse
• cross-model transfer

Innovation

• task vector transfer
• bilinear alignment
• training-free adaptation
• orthogonal Procrustes
• model transfer