Efficient Model Editing with Task Vector Bases: A Theoretical Framework and Scalable Approach

2025-02-03

AI Summary

Existing task vector editing methods lack theoretical foundations and suffer from high memory overhead, severely limiting scalability to multiple tasks. This paper proposes a scalable editing framework based on task vector bases. First, it establishes the first formal theoretical interpretation of task vector arithmetic. Second, it introduces a low-dimensional orthogonal basis to decompose and parameterize task vectors, achieving exponential memory compression. Third, it integrates weight-difference modeling with theory-driven geometric analysis to ensure compositional flexibility and strong generalization. Experiments demonstrate that, in multi-task editing scenarios, the proposed method reduces memory consumption by over 90% while matching the editing accuracy of full fine-tuning—and significantly outperforming existing heuristic approaches.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Computer Vision: Other Foundations of Computer Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Task vectors, which are derived from the difference between pre-trained and fine-tuned model weights, enable flexible task adaptation and model merging through arithmetic operations such as addition and negation. However, existing approaches often rely on heuristics with limited theoretical support, often leading to performance gaps comparing to direct task fine tuning. Meanwhile, although it is easy to manipulate saved task vectors with arithmetic for different purposes, such compositional flexibility demands high memory usage, especially when dealing with a huge number of tasks, limiting scalability. This work addresses these issues with a theoretically grounded framework that explains task vector arithmetic and introduces the task vector bases framework. Building upon existing task arithmetic literature, our method significantly reduces the memory cost for downstream arithmetic with little effort, while achieving competitive performance and maintaining compositional advantage, providing a practical solution for large-scale task arithmetic.

Problem

• Pre-trained Model Adaptation
• Task Vector Optimization
• Scalability and Memory Efficiency

Innovation

• Task Vector Arithmetic
• Memory Efficiency
• Scalable Solution