Task Singular Vectors: Reducing Task Interference in Model Merging

2024-11-26

AI Summary

Multi-task model merging often suffers from task interference, leading to performance degradation—particularly challenging when no additional training is permitted, making it difficult to balance accuracy and efficiency. To address this, we propose the Task-Specific Singular Vector (TSV) framework: (1) modeling task directions at the layer level via singular value decomposition (SVD) of layer-wise task matrices to extract low-rank TSV representations; (2) defining a task interference metric based on the angular distance between singular vectors; and (3) introducing two novel paradigms—TSV-C for parameter-efficient compression and TSV-M for low-coupling fusion. Our method retains only 10% of parameters while recovering 99% of the original multi-task accuracy, significantly outperforming mainstream approaches (e.g., Task Arithmetic) on standardized multi-task merging benchmarks. The framework enables quantifiable interference analysis, high-fidelity compression, and decoupled fusion within a unified theoretical and practical framework.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is susceptible to task interference. In this paper, we study task vectors at the layer level, focusing on task layer matrices and their singular value decomposition. In particular, we concentrate on the resulting singular vectors, which we refer to as Task Singular Vectors (TSV). Recognizing that layer task matrices are often low-rank, we propose TSV-Compress (TSV-C), a simple procedure that compresses them to 10% of their original size while retaining 99% of accuracy. We further leverage this low-rank space to define a new measure of task interference based on the interaction of singular vectors from different tasks. Building on these findings, we introduce TSV-Merge (TSV-M), a novel model merging approach that combines compression with interference reduction, significantly outperforming existing methods.

Problem

• Multi-task Learning
• Model Consolidation
• Performance Degradation

Innovation

• TSV-Merge
• TSV-Compress
• task singular vectors