Weight Weaving: Parameter Pooling for Data-Free Model Merging

2025-10-15

AI Summary

Conventional model ensembling requires tuning the scaling hyperparameter λ on a validation set, violating the “data-agnostic” assumption. Method: This paper proposes Weight Weaving—a data-free weight ensembling method that eliminates reliance on any training or validation data. It performs plug-and-play pooling (e.g., averaging or random sampling) over pretrained model weights within a predefined λ search space, replacing explicit hyperparameter optimization with structured weight aggregation. Contribution/Results: Modular and architecture-agnostic, Weight Weaving integrates seamlessly with ViTs and existing fusion strategies. Evaluated across multi-task learning, continual learning, and domain generalization on three vision transformers, it achieves up to +15.9 percentage points in average accuracy—demonstrating substantial gains in ensembling robustness and generalization without privileged data access.

Technology Category

• Machine Learning: Ensemble Methods
• Computer Vision: Multi-modal Vision
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Model merging provides a cost-effective and data-efficient combination of specialized deep neural networks through parameter integration. This technique leverages expert models across downstream tasks without requiring retraining. Most model merging approaches critically depend on scaling hyper-parameters $λ$, which weight each model's contribution globally or individually. Principled approaches for setting scaling factors without accessing any data (data-free) are scarce, often leading researchers to tune $λ$ using privileged data from the evaluation set, which is obviously unfeasible in practice. To address this limitation, we introduce Weight Weaving, a plug-and-play technique that pools model weights across $λ$ values search space using user-defined pooling functions, such as averaging, random selection, or even existing model merging methods. Our method demonstrates high modularity, imposing minimal constraints on the search space. It operates orthogonally to existing model merging methods and eliminates evaluation data requirements. We validate Weight Weaving across three ViT variants in three experimental setups: vision multi-task learning, vision continual learning, and domain generalization. Our method consistently improves the performance of several model merging methods, achieving average accuracy gains of up to 15.9 percentage points in a data-free setting.

Problem

• Addresses data-free hyper-parameter tuning in model merging
• Eliminates dependency on evaluation data for merging models
• Improves model merging performance without requiring training data

Innovation

• Weight Weaving pools model weights across lambda values
• Uses user-defined pooling functions like averaging or selection
• Eliminates data requirements for model merging techniques