Navigating the Accuracy-Size Trade-Off with Flexible Model Merging

2025-05-29

AI Summary

Model merging aims to achieve multi-task capability at zero training cost, yet existing methods suffer from either accuracy degradation or excessive deployment overhead. This paper proposes FlexMerge, the first data-free and scalable framework for flexible model merging. It introduces a progressive block-level fusion mechanism that enables on-demand generation of merged models of arbitrary size—from single-model to full-model configurations. Additionally, it proposes a data-agnostic model block serialization scheme coupled with a dynamic truncation strategy, ensuring compatibility with mainstream data-free merging algorithms such as Task Arithmetic. Evaluated across 30 vision and NLP tasks, FlexMerge achieves substantial performance gains with only marginal parameter increase: average accuracy improves by 4.2% for single-model merging, while deployment flexibility increases by an order of magnitude. Crucially, it is the first work to systematically characterize and controllably trade off accuracy versus parameter count along the Pareto frontier.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Model merging has emerged as an efficient method to combine multiple single-task fine-tuned models. The merged model can enjoy multi-task capabilities without expensive training. While promising, merging into a single model often suffers from an accuracy gap with respect to individual fine-tuned models. On the other hand, deploying all individual fine-tuned models incurs high costs. We propose FlexMerge, a novel data-free model merging framework to flexibly generate merged models of varying sizes, spanning the spectrum from a single merged model to retaining all individual fine-tuned models. FlexMerge treats fine-tuned models as collections of sequential blocks and progressively merges them using any existing data-free merging method, halting at a desired size. We systematically explore the accuracy-size trade-off exhibited by different merging algorithms in combination with FlexMerge. Extensive experiments on vision and NLP benchmarks, with up to 30 tasks, reveal that even modestly larger merged models can provide substantial accuracy improvements over a single model. By offering fine-grained control over fused model size, FlexMerge provides a flexible, data-free, and high-performance solution for diverse deployment scenarios.

Problem

• Balancing accuracy and model size in merging
• Reducing deployment costs of multiple fine-tuned models
• Achieving multi-task capabilities without expensive training

Innovation

• Flexible model merging for multi-task capabilities
• Data-free framework merging sequential blocks progressively
• Balances accuracy and size in diverse deployments