Learning to Recover Task Experts from a Multi-Task Merged Model

2026-06-25

AI Summary

This work addresses the performance degradation of task-specific experts in multitask model merging caused by parameter interference, as well as the high inference cost and storage overhead of existing dynamic methods that rely on redundant expert copies. The authors propose ReTeX, a framework that models parameter interference as an affine transformation of expert parameters and approximates it with a learnable additive offset, enabling a single merged model to recover near-original expert performance. Innovatively, ReTeX introduces a router-free task identifier that leverages singular value decomposition (SVD) subspace projection residuals to match task identities, achieving the first subspace-based task recognition without additional storage. Experiments demonstrate that ReTeX recovers over 95% of standalone expert performance across vision and NLP tasks and exhibits strong generalization and adaptive knowledge interpolation capabilities on unseen tasks.

Abstract

Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suffers from parameter interference. While dynamic merging models aim to bridge this gap, many works rely on the costly storage and loading of redundant expert components at inference. In this work, from the perspective of task expert, we view parameter interference as parameter perturbation introduced to each expert during merging process. We show that such parameter perturbations can be modeled as affine transformation, which can be approximated as additive offsets. Motivated by these, we propose Recover Task eXpert (ReTeX), a framework that predicts those offsets, in order to undo parameter interference and recover task-expert performance from a single merged checkpoint. To recover the appropriate expert when task identity is unknown, we introduce a router-free task identifier based on SVD subspace signatures computed offline before inference. At inference, the identifier selects the task whose subspace yields the smallest projection residual for a given input. As a result, ReTeX recovers over 95% of individual-expert performance in both vision and NLP domains, while significantly improving generalization to unseen tasks. Crucially, we also show that the parameter offset prediction leads to emergent adaptive interpolation of expert knowledge for out-of-distribution (OOD) tasks. ReTeX adaptively interpolates seen expert knowledge to handle unseen tasks. Our code is available at https://github.com/BAIKLAB/ReTeX

Problem

• multi-task model merging
• parameter interference
• task expert recovery
• inference efficiency
• out-of-distribution generalization

Innovation

• parameter interference
• affine transformation
• task expert recovery
• router-free task identification
• adaptive interpolation