HyperFix: Combinatorial Nonlinear Correction for Task Vector Merging

2026-08-11

AI Summary

Existing task vector merging methods rely on linear scaling and repeated hyperparameter tuning for specific task subsets, resulting in limited generalization. This work reframes task vector merging as a combinatorial nonlinear correction problem and introduces, for the first time, a composition-based nonlinear correction mechanism. Leveraging a lightweight hypernetwork, the approach learns conditional weight adjustments from singleton, pairwise, and triplet task subsets through local perturbation analysis and small-scale task updates. Trained only once, the method generalizes effectively to arbitrary task subsets, overcoming the constraints of linear merging. It achieves significant performance gains over current state-of-the-art approaches across multiple benchmarks while drastically reducing the need for repeated tuning, thereby enabling efficient and flexible task vector fusion.

Abstract

Task vectors enable model merging without joint retraining. In practice, the subset of task vectors to be merged may vary, but many existing methods use scalar tuning for a particular subset, requiring repeated tuning across subsets and restricting task vector merging to linear rescaling. We therefore formulate merging across varying task subsets as a combinatorial correction problem and introduce HyperFix, a lightweight hypernetwork that predicts subset-conditioned nonlinear corrections in weight space. Trained once on singleton, pair, and triple subsets from a task bank, HyperFix generalizes to larger subsets without per-subset optimization. Our local perturbation analysis bounds the residual correction beyond linear merging and motivates learning it from small task updates. Experiments across diverse benchmarks show that HyperFix outperforms existing task vector merging methods while reducing tuning cost.

Problem

• task vector merging
• combinatorial correction
• nonlinear correction
• subset generalization
• model merging

Innovation

• task vector merging
• nonlinear correction
• hypernetwork
• combinatorial optimization
• model merging