Task Addition and Weight Disentanglement in Closed-Vocabulary Models

2025-11-18

AI Summary

Closed-vocabulary image classification models pretrained without linguistic supervision struggle with efficient multi-task editing. Method: This work introduces task arithmetic—the first such application to language-free pretrained models—via a weight-decoupling–based task editing paradigm. Systematic experiments reveal that multiple vision Transformer pretraining mechanisms inherently support weight decoupling, providing a transferable structural foundation for task arithmetic. We integrate task addition with linear probe baselines and validate effectiveness on benchmarks including ImageNet. Results: Adding only a few task vectors enables high-accuracy, low-overhead task composition, significantly enhancing model flexibility and deployment efficiency in multi-task settings. This study demonstrates that pretraining universally facilitates task editing capability, thereby extending both the theoretical boundaries and practical applicability of task arithmetic in closed-vocabulary models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: (Large) Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Task arithmetic has recently emerged as a promising method for editing pre-trained extit{open-vocabulary} models, offering a cost-effective alternative to standard multi-task fine-tuning. However, despite the abundance of extit{closed-vocabulary} models that are not pre-trained with language supervision, applying task arithmetic to these models remains unexplored. In this paper, we deploy and study task addition in closed-vocabulary image classification models. We consider different pre-training schemes and find that extit{weight disentanglement} -- the property enabling task arithmetic -- is a general consequence of pre-training, as it appears in different pre-trained closed-vocabulary models. In fact, we find that pre-trained closed-vocabulary vision transformers can also be edited with task arithmetic, achieving high task addition performance and enabling the efficient deployment of multi-task models. Finally, we demonstrate that simple linear probing is a competitive baseline to task addition. Overall, our findings expand the applicability of task arithmetic to a broader class of pre-trained models and open the way for more efficient use of pre-trained models in diverse settings.

Problem

• Extending task arithmetic to closed-vocabulary image classification models
• Investigating weight disentanglement in pre-trained closed-vocabulary models
• Comparing task addition efficiency against linear probing baselines

Innovation

• Task arithmetic applied to closed-vocabulary models
• Weight disentanglement enables efficient task editing
• Linear probing competes with task addition performance