Task Alignment: A simple and effective proxy for model merging in computer vision

2026-04-14

AI Summary

This work addresses the challenge of merging multi-task vision models with heterogeneous decoders, a process traditionally hindered by costly hyperparameter tuning that scales poorly to complex tasks. The authors propose a lightweight, task-aligned proxy metric that, for the first time, enables efficient prediction of merged model performance without requiring downstream task evaluation. This approach dramatically accelerates hyperparameter selection, reducing tuning costs by several orders of magnitude while preserving model performance. Empirical results demonstrate that the framework effectively supports a range of complex vision tasks—including, but not limited to, CLIP-based classification—establishing a practical and scalable new paradigm for multi-task model fusion.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Efficiently merging several models fine-tuned for different tasks, but stemming from the same pretrained base model, is of great practical interest. Despite extensive prior work, most evaluations of model merging in computer vision are restricted to image classification using CLIP, where different classification datasets define different tasks. In this work, our goal is to make model merging more practical and show its relevance on challenging scenarios beyond this specific setting. In most vision scenarios, different tasks rely on trainable and usually heterogeneous decoders. Differently from previous studies with frozen decoders, where merged models can be evaluated right away, the non-trivial cost of decoder training renders hyperparameter selection based on downstream performance impractical. To address this, we introduce the task alignment proxy, and show how it can be used to speed up hyperparameter selection by orders of magnitude while retaining performance. Equipped with the task alignment proxy, we extend the applicability of model merging to multi-task vision models beyond CLIP-based classification.

Problem

• model merging
• computer vision
• multi-task learning
• decoder training
• hyperparameter selection

Innovation

• Task Alignment
• Model Merging
• Multi-task Vision
• Decoder Training
• Hyperparameter Selection